From 54592a5839f77840bacb17f8c295c379ab365432 Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 3 Aug 2026 02:50:38 -0400 Subject: [PATCH] docs: reorganize into DOC/ and simplify README - Move all docs (DEV, AGENT_Context, algorithm) into DOC/ via git mv - Rewrite root README as short intro + quick start + links into DOC/ - Add per-area reference docs: SCRIPTS (core scripts + libs + features), POS (pos CLI + compose config), APPS (picker + catalog), SYSTEMD (units) - Add DOC/README.md index; fix all cross-references --- .../AGENT_Context_Project.md | 22 +- DOC/APPS.md | 88 +++++++ DEV.md => DOC/DEV.md | 8 +- DOC/POS.md | 174 +++++++++++++ DOC/README.md | 20 ++ DOC/SCRIPTS.md | 188 ++++++++++++++ DOC/SYSTEMD.md | 92 +++++++ algorithm.md => DOC/algorithm.md | 0 README.md | 232 ++---------------- 9 files changed, 609 insertions(+), 215 deletions(-) rename AGENT_Context_Project.md => DOC/AGENT_Context_Project.md (95%) create mode 100644 DOC/APPS.md rename DEV.md => DOC/DEV.md (98%) create mode 100644 DOC/POS.md create mode 100644 DOC/README.md create mode 100644 DOC/SCRIPTS.md create mode 100644 DOC/SYSTEMD.md rename algorithm.md => DOC/algorithm.md (100%) diff --git a/AGENT_Context_Project.md b/DOC/AGENT_Context_Project.md similarity index 95% rename from AGENT_Context_Project.md rename to DOC/AGENT_Context_Project.md index 49ed3be..7a8288e 100644 --- a/AGENT_Context_Project.md +++ b/DOC/AGENT_Context_Project.md @@ -94,8 +94,18 @@ Linux_post_install/ │ ├── autostart.service # Runs autostart.sh on boot │ └── ssh-agent.service # System-wide SSH agent socket │ -├── README.md # User-facing documentation -├── DEV.md # Developer guide +├── README.md # User-facing intro + quick start (links into DOC/) +│ +├── DOC/ # All documentation +│ ├── README.md # Docs index +│ ├── SCRIPTS.md # Installer scripts, libs, features — reference +│ ├── POS.md # pos CLI reference +│ ├── APPS.md # Optional apps reference +│ ├── SYSTEMD.md # Systemd units + completion +│ ├── DEV.md # Developer guide +│ ├── AGENT_Context_Project.md # This file — AI agent context +│ └── algorithm.md # Algorithm diagrams +│ ├── .gitignore # Excludes secrets, Python artifacts, OS files └── .gitmodules # Submodule: compose/scale-tail → ScaleTail ``` @@ -298,7 +308,7 @@ ScaleTail provides 119+ Docker Compose templates with a Tailscale sidecar patter ### Adding a New App -1. Create `apps/.sh` following the template in DEV.md +1. Create `apps/.sh` following the template in DOC/DEV.md 2. It auto-appears in the interactive picker — no registration needed --- @@ -374,7 +384,7 @@ System-wide flag store at `/usr/local/share/linux_post_install/flags/`: ### Adding a New App -1. Create `apps//.sh` following the template in DEV.md +1. Create `apps//.sh` following the template in DOC/DEV.md 2. It auto-appears in the interactive picker — no registration needed ### Adding a New Tool @@ -382,7 +392,7 @@ System-wide flag store at `/usr/local/share/linux_post_install/flags/`: 1. Create `bin/pos--` following conventions 2. Add system deps to `PACKAGES` array in `preinstall.sh` (if needed) 3. Add config logic to `postinstall.sh` (if needed, with `.gitignore` for secrets) -4. Update `README.md` +4. Update `DOC/POS.md` (and root `README.md` only if the category list changes) 5. Test: `bash -n bin/your-tool && shellcheck bin/your-tool` ### Testing @@ -453,5 +463,5 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | Modify UFW/firewall logic | Edit `bin/pos-system-firewall` | | Modify pos logging | Edit log setup in `bin/pos` | | Modify install phases/flags | Edit arg parsing in `install.sh` | -| Update documentation | Edit `README.md` and/or `DEV.md` | +| Update documentation | Edit the relevant doc under `DOC/` (index: `DOC/README.md`) | | Add a secret config file | Add to `config/`, update `.gitignore`, add copy logic in `postinstall.sh` | diff --git a/DOC/APPS.md b/DOC/APPS.md new file mode 100644 index 0000000..f108da9 --- /dev/null +++ b/DOC/APPS.md @@ -0,0 +1,88 @@ +# Optional Apps Reference + +`apps/` holds 15 optional desktop application installers, one script per app in `apps//.sh`. They are **not** installed by the core bootstrap — run the picker explicitly. + +- [The picker — `apps/install.sh`](#the-picker--appsinstallsh) +- [How an app installer works](#how-an-app-installer-works) +- [App catalog](#app-catalog) + +--- + +## The picker — `apps/install.sh` + +**Purpose:** discover every app under `apps/` and install/uninstall the selection. Apps are auto-discovered from the directory structure — no registration step. + +### Usage + +```bash +bash apps/install.sh # interactive install selection +bash apps/install.sh --all # install everything +bash apps/install.sh brave vscode # install specific apps +bash apps/install.sh --uninstall # interactive uninstall selection +bash apps/install.sh --uninstall --all # uninstall everything +bash apps/install.sh --uninstall brave # uninstall a specific app +``` + +(Also reachable via `./install.sh --apps` / `--full`.) + +### How it works + +1. Parses `--all`, `--uninstall`, and positional app names. +2. Scans `apps//*.sh` to build the catalog (skips non-app dirs). +3. Picks apps three ways: named on the command line (unknown names are skipped with a warning), `--all`, or an interactive y/n picker grouped by category. +4. Runs `bash apps//.sh [uninstall]` for each selected app, with a progress header and an overall elapsed-time banner. + +### Configuration + +- Categories: `browsers`, `development`, `media`, `networking`, `remote-access`, `system`, `utilities`. +- Adding an app = dropping `apps//.sh` into the folder. See [DEV.md](DEV.md) for the required installer conventions. + +--- + +## How an app installer works + +Every app script follows the same shape: + +```bash +install_() { … } # idempotent: checks command -v (or flatpak list) first +uninstall_() { … } # also idempotent; purges and removes any added repos/keys +case "${1:-}" in + uninstall) uninstall_ ;; + *) install_ ;; +esac +``` + +Installation methods used across the catalog: + +| Method | Example | +|--------|---------| +| `apt` package | `sudo apt install -y obs-studio` | +| Official installer script | `curl -fsSL https://tailscale.com/install.sh \| sh` | +| Custom apt repo (added at install, removed at uninstall) | Brave, VS Code | +| `.deb` file | `curl` → `dpkg -i` → `apt-get install -f -y` | +| GitHub release archive | scrcpy (tar.gz → `/usr/local/lib/`) | +| AppImage | AFFiNE (`/opt/affine` + desktop entry) | +| Flatpak | LocalSend (`flatpak install -y flathub …`) | + +--- + +## App catalog + +| App | Category | What it is | Install method | +|-----|----------|------------|----------------| +| Brave | browsers | Brave browser | apt repo + `apt install brave-browser` | +| opencode | development | AI coding agent | official script → `~/.opencode/bin` | +| VS Code | development | Code editor | Microsoft apt repo + `apt install code` | +| OBS Studio | media | Screen recording / streaming | `apt install obs-studio` | +| scrcpy | media | Android mirror/control | GitHub release (latest) → `/usr/local/lib/scrcpy-` + desktop entry | +| VLC | media | Media player | `apt install vlc` | +| NetBird | networking | Mesh VPN | official script; join with `sudo netbird up --setup-key ` | +| Tailscale | networking | WireGuard-based VPN | official script; start with `sudo tailscale up` | +| ZeroTier | networking | Virtual LAN | official script; join with `sudo zerotier-cli join ` | +| Termius | remote-access | SSH client | `.deb` from termius.com | +| VNC Viewer | remote-access | VNC client (TigerVNC) | `apt install tigervnc-viewer` | +| Docker Engine | system | Container runtime | get.docker.com; adds user to `docker` group (re-login needed) | +| QEMU + KVM | system | Virtualization + virt-manager | `apt install` (qemu-system, libvirt, bridge-utils, virt-manager); adds user to `libvirt`/`kvm` groups | +| AFFiNE | utilities | Knowledge base (AppImage) | GitHub release → `/opt/affine` + desktop entry | +| btop | utilities | Resource monitor | `apt install btop` | +| LocalSend | utilities | Local file sharing | flatpak (installs flatpak + flathub if missing) | diff --git a/DEV.md b/DOC/DEV.md similarity index 98% rename from DEV.md rename to DOC/DEV.md index c5d9e80..6b14039 100644 --- a/DEV.md +++ b/DOC/DEV.md @@ -119,15 +119,15 @@ PACKAGES=( ### 3. Add config files (if needed) -Place defaults in `config/` and add copy logic to `postinstall.sh`. If they contain secrets, add to `.gitignore` and document in README. +Place defaults in `config/` and add copy logic to `postinstall.sh`. If they contain secrets, add to `.gitignore` and document in `DOC/`. ### 4. Add SSH keys (if needed) Place public keys in `config/authorized_keys` (one per line). `postinstall.sh` reads this file automatically. -### 5. Update README +### 5. Update the docs -Add a section under the relevant category in README.md. +Add a section for the new command in `DOC/POS.md`. ### 6. Test @@ -188,7 +188,7 @@ Place it in `apps//.sh`. It auto-appears in the picker — no re 1. Find the script in `bin/` 2. Understand its contract (args, output, exit codes) 3. Make the change — keep it idempotent -4. Update README if behaviour changed +4. Update `DOC/POS.md` (or the relevant doc) if behaviour changed 5. Run `shellcheck` on the modified file --- diff --git a/DOC/POS.md b/DOC/POS.md new file mode 100644 index 0000000..99d1f42 --- /dev/null +++ b/DOC/POS.md @@ -0,0 +1,174 @@ +# `pos` CLI Reference + +`pos` is the unified command-line interface installed to `/usr/local/bin/`. Every tool is a small script in `bin/` with a `pos--` name. This document explains the dispatcher and every command. + +- [The dispatcher — `bin/pos`](#the-dispatcher--binpos) +- [Logging behavior](#logging-behavior) +- [Commands](#commands) + - [network](#network) + - [docker](#docker) + - [media](#media) + - [system](#system) + - [ssh](#ssh) + - [vbox](#vbox) + - [flags](#flags) +- [Legacy wrappers](#legacy-wrappers) + +--- + +## The dispatcher — `bin/pos` + +**Purpose:** turn `pos [args]` into a call to the matching `pos-*` script. + +### How it works + +`pos` scans its own directory for executable `pos-*` files and tries **variable-length argument matching**, longest first. For `pos docker compose up jellyfin`: + +``` +tries pos-docker-compose-up-jellyfin (not found) +tries pos-docker-compose-up (not found) +tries pos-docker-compose (found) → runs with args "up jellyfin" +``` + +`pos help ` runs ` --help`. Running `pos` with no args prints the built-in usage text (which doubles as the category cheat-sheet). + +--- + +## Logging behavior + +Every non-interactive `pos` invocation logs to `~/.local/share/linux_post_install/logs/`: + +- Per-command files: `YYYYMMDD_HHMMSS_pos_.log` (full stdout + stderr). +- `pos.log`: one line per invocation — command, log file, exit code. +- **Interactive** commands (`pos system firewall`, `pos media mp4`) only log the invocation, not their output. + +--- + +## Commands + +### network + +| Command | File | Purpose | Configuration | +|---------|------|---------|---------------| +| `pos network ip` | `bin/pos-network-ip` | Show interfaces, default route, public IP | None. Public IP via `https://ifconfig.me` (5s timeout) | +| `pos network checkport ` | `bin/pos-network-checkport` | Check if a TCP port is open | None. Uses `/dev/tcp` with a 2s timeout; exit 0/1 via OPEN/CLOSED | +| `pos network scan [--full] [--retries N]` | `bin/pos-network-scan` | Two-phase nmap scan | See below | + +**`pos network scan` in detail:** + +- Phase 1 — fast host discovery (`nmap -sn -T5`), prints the live host list. +- Phase 2 (only with `--full`) — service/version scan (`-sV -sC`), plus OS detection and NSE scripts if run with privileges; shows ports, OS, SSH host keys, HTTP titles, NetBIOS/SMB info. +- Accepts a bare IP (treated as `/32`) or a CIDR. +- Auto-raises to `sudo nmap` when possible (root, passwordless sudo, or an interactive terminal with `--full`). +- `--retries N` tunes discovery retries (default 1). + +### docker + +| Command | File | Purpose | Configuration | +|---------|------|---------|---------------| +| `pos docker ps` | `bin/pos-docker-ps` | Enhanced container list: name, image, health, uptime, IPs, ports, ID, plus a healthy/unhealthy summary | None. Requires Docker + Python 3 | +| `pos docker health` | `bin/pos-docker-health` | One-glance health dashboard; **exits 1** if any container is unhealthy | None. Checks all containers including stopped ones | +| `pos docker compose …` | `bin/pos-docker-compose` | ScaleTail service manager | See [Docker Compose / ScaleTail](#docker-compose--scaletail) below | + +#### Docker Compose / ScaleTail + +**`pos docker compose ls`** — list available ScaleTail service templates. + +**`pos docker compose installed`** — list deployed services under `$SERVICES_BASE`. + +**`pos docker compose up `** — deploy a service: + +1. If not yet deployed, creates `$SERVICES_BASE//` with `config/` and `data/`, copies the template's `compose.yaml`. +2. If no `.env` exists, copies the template's `.env` (or writes a default) and fills in your global config values (`TS_AUTHKEY`, `TZ`, `DNS_SERVER`). +3. If `TS_AUTHKEY` is still empty, prompts for it. +4. Offers to edit `.env` before starting (default **yes** on first deploy). +5. Runs `docker compose up -d`. + +**`pos docker compose down/restart/logs `** — stop, restart, or tail logs of a deployment. + +**`pos docker compose update`** — `git pull` the ScaleTail templates, then refresh the `compose.yaml` of every deployed service. **Per-service `.env` files are never touched.** + +**`pos docker compose config [show]`** — show the global config file and `SERVICES_BASE`. + +**`pos docker compose config set KEY=VALUE`** — set/update a global default in `~/.config/linux_post_install/compose.env`. + +**`pos docker compose config edit`** — open the global config in `$EDITOR` (creates a default file first). + +Configuration (three layers, most specific wins): + +| Layer | File | Notes | +|-------|------|-------| +| Template defaults | `/usr/local/share/linux_post_install/scale-tail/services//.env` | Read-only | +| Global config | `~/.config/linux_post_install/compose.env` | Edited via `config set` / `config edit` | +| Per-service | `$SERVICES_BASE//.env` | Created on first `up`, **never overwritten** | + +Global config keys: + +| Key | Required | Default | Purpose | +|-----|----------|---------|---------| +| `TS_AUTHKEY` | yes | — | Tailscale auth key for the sidecar | +| `TZ` | no | `Europe/Amsterdam` | Service timezone | +| `DNS_SERVER` | no | `9.9.9.9` | DNS server | +| `SERVICES_BASE` | no | `/srv` | Deployment root | + +### media + +| Command | File | Purpose | Configuration | +|---------|------|---------|---------------| +| `pos media mp3 ` | `bin/pos-media-mp3` | Download audio as MP3 via yt-dlp, with thumbnail + metadata | Output to `~/Music/%(title)s.%(ext)s`, `--audio-quality 0` | +| `pos media mp4 ` | `bin/pos-media-mp4` | Download video via yt-dlp with **interactive format selection** | Lists formats (`yt-dlp -F`), asks for a format ID, saves to `~/Videos/` | + +### system + +| Command | File | Purpose | Configuration | +|---------|------|---------|---------------| +| `sudo pos system firewall` | `bin/pos-system-firewall` | Interactive UFW ("UFW POWER") menu: add/delete rules, status, enable/disable/reset, default policies | Must run as root. Every command is previewed and confirmed before execution; supports `--dry-run`; keeps a history of executed commands | + +### ssh + +| Command | File | Purpose | Configuration | +|---------|------|---------|---------------| +| `pos ssh load-keys` | `bin/pos-ssh-load-keys` | Load all `~/.ssh/id_*` private keys into the ssh-agent | Uses `SSH_AUTH_SOCK` (default `/run/ssh-agent/socket`, provided by `ssh-agent.service`); skips `.pub`, `known_hosts`, `authorized_keys`, `config`; validates keys before adding | + +### vbox + +**File:** `bin/pos-vbox` +**Purpose:** manage disposable Docker containers as lightweight "VMs". Each container gets a bind-mounted host directory so files persist after the container is removed. Containers carry the label `linux_post_install.vbox=true`. + +| Command | Behavior | +|---------|----------| +| `pos vbox create [image] [--dir ]` | Creates a container from `ubuntu:22.04` (or the given image), bind-mounting `~/` (or `--dir`, or `.` for cwd) as the working directory; prompts to enter immediately | +| `pos vbox enter ` | Shell into the container (auto-starts it if stopped); detects the working dir from the container mounts | +| `pos vbox start/stop/rm ` | Start, stop, or force-remove the container | +| `pos vbox ls` | List vbox containers only (label filter) | + +### flags + +Feature-flag management CLIs (see [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags)): + +| Command | Purpose | +|---------|---------| +| `flag-reader` | List all flags + status (`set: ` / `unset: `) | +| `flag-reader ` | Check one flag; exit 0 if set, 1 if not | +| `flag-reader --raw ` | Print only the stored value (script-friendly) | +| `flag-set [value]` | Set a flag, optionally with a value (requires sudo) | +| `flag-clear ` | Unset a flag (requires sudo) | + +--- + +## Legacy wrappers + +Thin 2-line scripts that `exec pos … "$@"`. All of them still work: + +| Wrapper | Forwards to | +|---------|-------------| +| `wr-ip` | `pos network ip` | +| `wr-checkport` | `pos network checkport` | +| `wr-scan-ping` | `pos network scan` | +| `wr-docker` | `pos docker` | +| `wr-compose` | `pos docker compose` | +| `wr-ufw` | `pos system firewall` | +| `mp3` | `pos media mp3` | +| `mp4` | `pos media mp4` | +| `vbox` | `pos vbox` | +| `ssh-load-all` | `pos ssh load-keys` | diff --git a/DOC/README.md b/DOC/README.md new file mode 100644 index 0000000..6d1fda9 --- /dev/null +++ b/DOC/README.md @@ -0,0 +1,20 @@ +# Documentation + +Everything in this folder is reference material for the `Linux_post_install` project. The root [README](../README.md) is the short intro + quick start; this folder holds the detail. + +| Document | What it covers | +|----------|----------------| +| [SCRIPTS.md](SCRIPTS.md) | Core installer scripts: `install.sh`, `preinstall.sh`, `postinstall.sh`, `lib/common.sh`, `lib/flags.sh`, `features/autostart.sh` — purpose, how each works, configuration | +| [POS.md](POS.md) | The `pos` CLI: dispatcher, every `pos-*` command, Docker Compose / ScaleTail config, legacy wrappers, flag CLIs | +| [APPS.md](APPS.md) | Optional apps: `apps/install.sh` picker, installer conventions, full app catalog | +| [SYSTEMD.md](SYSTEMD.md) | Systemd units (`autostart.service`, `ssh-agent.service`), feature-flag gating, bash completion | +| [DEV.md](DEV.md) | Developer guide: architecture, conventions, how to add tools/apps/features, commit guidelines | +| [AGENT_Context_Project.md](AGENT_Context_Project.md) | Single-source context doc for AI agents working on the repo | +| [algorithm.md](algorithm.md) | ASCII diagrams: install flow, `pos` dispatch, compose `up`, config cascade, logging, vbox lifecycle | + +## Quick navigation + +- Just installed and want to use it? → [POS.md](POS.md) +- Adding a package? → [SCRIPTS.md → preinstall.sh](SCRIPTS.md#preinstallsh--system-packages) +- Adding a CLI tool or app? → [DEV.md](DEV.md) +- First deploy of a self-hosted service? → [POS.md → Docker Compose](POS.md#docker-compose--scaletail) diff --git a/DOC/SCRIPTS.md b/DOC/SCRIPTS.md new file mode 100644 index 0000000..94292da --- /dev/null +++ b/DOC/SCRIPTS.md @@ -0,0 +1,188 @@ +# Core Scripts Reference + +Everything that runs during the bootstrap install: `install.sh`, `preinstall.sh`, `postinstall.sh`, the shared libraries, and `features/`. For the `pos` CLI tools see [POS.md](POS.md), for apps see [APPS.md](APPS.md), for services see [SYSTEMD.md](SYSTEMD.md). + +--- + +## Table of contents + +- [install.sh — the orchestrator](#installsh--the-orchestrator) +- [preinstall.sh — system packages](#preinstallsh--system-packages) +- [postinstall.sh — user configuration](#postinstallsh--user-configuration) +- [lib/common.sh — shared library](#libcommonsh--shared-library) +- [lib/flags.sh — feature flags](#libflagssh--feature-flags) +- [features/autostart.sh — boot-time feature](#featuresautostartsh--boot-time-feature) + +--- + +## install.sh — the orchestrator + +**File:** `install.sh` (run as `./install.sh`) +**Purpose:** the entry point. Coordinates all four install phases and the optional apps/features installs. + +### How it works + +1. **Pre-parse `--no-color`** before anything else, so colors are disabled early (`TERM=dumb` is exported). +2. Source `lib/common.sh` (logging, `run`, `spawn`, …) and `lib/flags.sh` (feature flags). +3. Parse CLI options. +4. For each phase, `should_run ` decides whether to run it: + - `--skip ` removes a phase (takes precedence). + - `--steps ` restricts the run to the listed phases only (`1,3,4` or `1-3`). + - Phase map: `1=preinstall`, `2=scripts`, `3=postinstall`, `4=scalepoint` (+ `apps` handled separately). + +The phases: + +| # | Phase | Script/action | +|---|-------|----------------| +| 1 | preinstall | `preinstall.sh` — apt packages + yt-dlp | +| 2 | scripts | Copies `bin/*` → `/usr/local/bin/` (755), `lib/common.sh` + `lib/flags.sh` → `/usr/local/bin/` (644). With `--feature`: also installs `features/*` (see below) | +| 3 | postinstall | `postinstall.sh` — PATH, completion, SSH keys, systemd | +| 4 | scalepoint | Shallow-clones ScaleTail templates to `/usr/local/share/linux_post_install/scale-tail` | +| 5 (opt) | apps | `apps/install.sh` when `--apps` (interactive) or `--full` (all, non-interactive) | + +**Feature block (Phase 2, only with `--feature`):** for every file in `features/` it copies it to `/usr/local/bin/`. If the destination already exists it asks **"Overwrite existing …? [y/N]"** (default keeps your file), then always sets the feature flag via `flag_set` (name derived as ``). + +### Configuration + +No config file — everything is command-line: + +| Option | Effect | +|--------|--------| +| `--apps` | Run the interactive app picker after core install | +| `--full` | Core install + every app (non-interactive) | +| `--feature` | Install `features/` scripts to `/usr/local/bin/` (prompts on overwrite), sets their flags | +| `--dry-run` | Log every action instead of executing. **Note:** applies to `install.sh` itself; `postinstall.sh` runs as a subprocess and does not inherit `DRY_RUN` | +| `--skip ` | Skip a phase (repeatable): `preinstall`, `scripts`, `postinstall`, `scalepoint`, `apps` | +| `--steps ` | Run only listed phases: `1,3,4` or `1-3` | +| `--no-color` | Disable colored output | +| `-h`, `--help` | Show usage | + +--- + +## preinstall.sh — system packages + +**File:** `preinstall.sh` +**Purpose:** Phase 1 — installs the base system packages and yt-dlp. +**Run:** automatically by `install.sh`, or standalone with `--dry-run`. + +### How it works + +1. `apt update`. +2. Installs the package list. +3. Downloads the latest `yt-dlp` binary to `/usr/local/bin/yt-dlp` and makes it executable. +4. Verifies a couple of tools (`git --version`, `yt-dlp --version`). + +### Configuration + +The package list is the `PACKAGES` array: + +```bash +PACKAGES=( + git curl wget vim nano tmux tree jq + unzip zip rsync htop btop telnet + net-tools iputils-ping traceroute tcpdump nmap + openssh-client openssh-server ufw fail2ban + ca-certificates gnupg lsb-release + python3 python3-pip rclone +) +``` + +Add or remove package names here. `nmap` and `fail2ban` are used later by `pos network scan` and `postinstall.sh`. + +--- + +## postinstall.sh — user configuration + +**File:** `postinstall.sh` (runs as your user) +**Purpose:** Phase 3 — configures the user environment, SSH keys, and systemd services. + +### How it works + +1. **rclone config** — if `config/rclone.conf` exists (gitignored), installs it to `~/.config/rclone/rclone.conf` (600). +2. **PATH** — appends a `PATH` line to `~/.bashrc` if not already present. +3. **pos bash completion** — installs `completions/pos.bash` to `/usr/local/share/bash-completion/completions/` and sources it from `~/.bashrc`. +4. **SSH authorized keys** — if `config/authorized_keys` exists, appends missing keys to `~/.ssh/authorized_keys` (skips comments and duplicates, chmod 600). +5. **systemd services** — copies `systemd/*.service` to `/etc/systemd/system/`, daemon-reloads, then enables each service. **`autostart.service` is only enabled when the `autostart` feature flag is set** (see [lib/flags.sh](#libflagssh--feature-flags)); otherwise it's skipped with a hint to run `./install.sh --feature`. + +### Configuration + +- SSH keys: `config/authorized_keys` (one per line, gitignored). +- rclone config: `config/rclone.conf` (gitignored). +- The PATH line and completion line are embedded strings at the top of the file — edit there to change them. +- The `autostart` flag (set by `./install.sh --feature`) controls whether `autostart.service` gets enabled. + +--- + +## lib/common.sh — shared library + +**File:** `lib/common.sh` (installed to `/usr/local/bin/common.sh`) +**Purpose:** colors, logging, timers, spinners, dry-run-aware execution, and prompts. Sourced by most scripts. + +### How it works + +Auto-disables colors when stdout is not a TTY. The `run` helper is the dry-run hook: scripts that want `--dry-run` support run every side-effecting command through `run`. + +### Configuration / API + +| Function | Purpose | +|----------|---------| +| `log "msg"` | Green `[+]` status line | +| `warn "msg"` | Yellow `[!]` warning | +| `err "msg"` | Red `ERROR:` line to stderr, then `exit 1` | +| `ok "msg"` | Green `OK` prefix line | +| `section "title"` | Cyan-bordered section header | +| `step N T "msg"` | Numbered step header (`[N/T] msg`) | +| `run cmd…` | Executes the command, or logs `(dry-run)` when `DRY_RUN=1` | +| `spawn "msg" cmd…` | Runs with an animated spinner + elapsed time; prints captured stderr and exits on failure | +| `timer_start` / `timer_stop` | Track and print elapsed time | +| `confirm "prompt" [default]` | Yes/no prompt; default `y` (`[Y/n]`) unless `n` given (`[y/N]`) | + +--- + +## lib/flags.sh — feature flags + +**File:** `lib/flags.sh` (installed to `/usr/local/bin/flags.sh`) +**Purpose:** a system-wide, per-feature flag store. Flags mark features as installed/opted-in and gate behavior (e.g. systemd enablement) elsewhere. + +### How it works + +One file per flag in `$FLAGS_DIR`. **Presence = set, file content = optional value.** Reads are plain file ops; writes go through `run` + `sudo` so they respect `--dry-run`. Installed by `./install.sh --feature`; also usable directly: + +```bash +source lib/flags.sh +flag_set autostart # bare flag +flag_set app "2.1" # flag with a value +flag_is_set autostart # 0 if set, 1 if not +flag_value app # prints "2.1" +flag_list # names of all set flags +flag_clear autostart +``` + +### Configuration + +| Setting | Location | +|---------|----------| +| `FLAGS_DIR` (env) | Default `/usr/local/share/linux_post_install/flags` (dir 755, files 644). Overridable via environment for testing | +| CLI wrappers | `flag-reader`, `flag-set`, `flag-clear` (see [POS.md](POS.md)) | + +--- + +## features/autostart.sh — boot-time feature + +**File:** `features/autostart.sh` (installed to `/usr/local/bin/autostart.sh` by `./install.sh --feature`) +**Purpose:** runs once at boot via `autostart.service` (only when the `autostart` flag is green) and logs basic connectivity status. + +### How it works + +Appends timestamped lines to `~/.autostart.log`: + +``` +[] autostart running +[] Network: online # ping 8.8.8.8 succeeded +[] autostart complete +``` + +### Configuration + +- Log file: `$HOME/.autostart.log` (edit the `LOG` variable at the top). +- The script is the one you're *most* likely to customize — this is exactly why it lives in `features/` instead of `bin/`: a plain reinstall never overwrites your edits. diff --git a/DOC/SYSTEMD.md b/DOC/SYSTEMD.md new file mode 100644 index 0000000..3567e10 --- /dev/null +++ b/DOC/SYSTEMD.md @@ -0,0 +1,92 @@ +# Systemd & Shell Integration Reference + +The units installed and enabled by `postinstall.sh`, plus the `pos` bash completion. + +- [Services](#services) + - [`autostart.service`](#autostartservice) + - [`ssh-agent.service`](#ssh-agentservice) +- [Feature-flag gating](#feature-flag-gating) +- [Bash completion](#bash-completion) + +--- + +## Services + +`postinstall.sh` copies every `systemd/*.service` to `/etc/systemd/system/`, runs `systemctl daemon-reload`, then enables each one (see the gating rule below). + +### autostart.service + +**Purpose:** run `features/autostart.sh` at boot (after the network is online) and keep retrying if it fails. + +```ini +[Unit] +Description=My Linux Autostart Script +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/local/bin/autostart.sh +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target +``` + +**Configuration:** point `ExecStart` at your boot script. Because the target script is a *feature*, this unit is only **enabled** when the `autostart` flag is set — the file is still copied, but a skipped feature leaves the unit present-but-disabled. + +### ssh-agent.service + +**Purpose:** a system-wide SSH agent, one shared socket for all sessions (so `pos ssh load-keys` and everyday ssh work without per-login agents). + +```ini +[Unit] +Description=SSH Authentication Agent +After=network.target + +[Service] +Type=simple +ExecStartPre=mkdir -p /run/ssh-agent +ExecStart=/usr/bin/ssh-agent -D -a /run/ssh-agent/socket +ExecStartPost=/bin/sh -c 'chmod 666 /run/ssh-agent/socket' +ExecStopPost=/bin/sh -c 'rm -f /run/ssh-agent/socket' +Restart=on-failure + +[Install] +WantedBy=multi-user.target +``` + +**Configuration:** socket at `/run/ssh-agent/socket` (world-readable/writable). `~/.bashrc` (set by `postinstall.sh`) exports `SSH_AUTH_SOCK` to it. Not gated on any feature flag. + +--- + +## Feature-flag gating + +The systemd loop in `postinstall.sh` special-cases `autostart.service`: + +```bash +if [ "$svc_name" = "autostart.service" ] && ! flag_is_set autostart; then + warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)" + continue +fi +``` + +Set the flag with `./install.sh --feature` (or `flag-set autostart`). See [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags). + +--- + +## Bash completion + +**File:** `completions/pos.bash` +**Purpose:** tab-completion for the `pos` CLI. + +### How it works + +- **Dynamically discovers** subcommands by listing executable `pos-*` files next to the `pos` binary — no hard-coded command list, so new tools complete automatically. +- Works with the `bash-completion` package (`_init_completion`) and falls back to a manual init if it isn't loaded. +- Provides completion for the first two words of `pos `. + +### Configuration + +Installed by `postinstall.sh` to `/usr/local/share/bash-completion/completions/pos.bash` and sourced from `~/.bashrc`. To load it manually: `source completions/pos.bash` (or copy into `/etc/bash_completion.d/`). diff --git a/algorithm.md b/DOC/algorithm.md similarity index 100% rename from algorithm.md rename to DOC/algorithm.md diff --git a/README.md b/README.md index be2eaa3..ea456d5 100755 --- a/README.md +++ b/README.md @@ -2,22 +2,20 @@ > One command turns a bare Debian/Ubuntu install into a fully productive machine. ---- +## What is this -## What Is This +After reinstalling Linux you usually need to install packages, set up SSH, configure a firewall, and install apps. This repo automates all of that in one go. -After reinstalling Linux, you usually need to install packages, set up SSH, configure the firewall, and install apps. This repo automates all of that in one go. +It is a **personal toolkit** — a bootstrap script, a unified `pos` CLI for everyday tasks, optional app installers, and self-hosted services via ScaleTail + Tailscale. **What you get:** -- **25+ system packages** installed automatically (git, curl, tmux, ufw, fail2ban, etc.) -- **Unified `pos` CLI** — one command for network, Docker, media, system, and SSH tasks -- **119+ self-hosted services** via ScaleTail + Tailscale (Jellyfin, Home Assistant, etc.) -- **15 optional desktop apps** (VS Code, Docker Desktop, Brave, OBS, etc.) — pick what you want -- **systemd services** for SSH agent and boot-time automation -- **Everything in `/usr/local/bin/`** — you can delete the repo after install - ---- +- 25+ system packages installed automatically +- The `pos` CLI: network, Docker, media, system, SSH, and vbox tools +- 15 optional desktop apps (VS Code, Brave, OBS, Tailscale, …) — pick what you want +- 119+ self-hosted services with Tailscale access (Jellyfin, Home Assistant, …) +- systemd services for SSH agent and boot-time automation +- Everything lands in `/usr/local/bin/` — you can delete the repo after install ## Quick Start @@ -25,204 +23,28 @@ After reinstalling Linux, you usually need to install packages, set up SSH, conf git clone https://gitea.skink-platy.ts.net/admin/Linux_post_install.git cd Linux_post_install ./install.sh # core: packages + CLI + services + ScaleTail -./install.sh --apps # core + interactive app picker -./install.sh --full # core + all apps (non-interactive) -./install.sh --feature # core + install features/ scripts (prompts on overwrite) +./install.sh --feature # also install features/ scripts (asks before overwriting) +./install.sh --apps # also install optional desktop apps (interactive) ``` -**Flags:** - | Flag | What it does | |------|-------------| -| `--apps` | Run interactive app picker after core install | -| `--full` | Core install + all apps (no prompts) | -| `--feature` | Install `features/` scripts to `/usr/local/bin/` (asks before overwriting), sets their feature flags | -| `--dry-run` | Preview without executing anything | -| `--skip ` | Skip a phase (repeatable): `preinstall`, `scripts`, `postinstall`, `scalepoint` | -| `--steps ` | Run specific phases only, e.g. `--steps 1,3` or `--steps 1-3` | +| `--feature` | Install `features/` scripts to `/usr/local/bin/`, sets their flags | +| `--apps` | Interactive app picker after core install | +| `--full` | Core install + all apps (non-interactive) | +| `--dry-run` | Preview without executing | +| `--skip ` | Skip a phase: `preinstall`, `scripts`, `postinstall`, `scalepoint`, `apps` | +| `--steps ` | Run only specific phases, e.g. `--steps 1,3` | | `--no-color` | Disable colored output | ---- +## Documentation -## What Gets Installed - -| Phase | Script | What happens | -|-------|--------|-------------| -| 1 | `preinstall.sh` | `apt update` + 25+ packages + yt-dlp + fail2ban | -| 2 | `install.sh` | Copies all `bin/` tools to `/usr/local/bin/` | -| 3 | `postinstall.sh` | Configures fail2ban, SSH agent, PATH, bash completion, systemd services | -| 4 | ScaleTail clone | Downloads 119+ Docker Compose templates with Tailscale sidecar | -| 5 (opt) | `apps/install.sh` | Installs desktop apps you select | -| opt | `./install.sh --feature` | Installs `features/` scripts (never overwrites without asking) | - ---- - -## Features & Flags - -`features/` holds scripts you're likely to customize (like `autostart.sh`), kept out of `bin/` so a plain re-install never resets them. - -```bash -./install.sh --feature # install features/ — asks before overwriting -``` - -- Each feature is copied to `/usr/local/bin/`; if the file already exists you're asked **"Overwrite? [y/N]"** — your existing config is kept by default. -- A successful install sets a **feature flag** at `/usr/local/share/linux_post_install/flags/` (presence = set, file content = optional value). -- Flags drive systemd: e.g. `autostart.service` is enabled only when the `autostart` flag is green. - -Inspect and manage flags: - -```bash -flag-reader # list all flags + status -flag-reader autostart # check one flag (exit 0 if set) -flag-reader --raw autostart # print the raw value only (script-friendly) -flag-set autostart prod # set a flag, optionally with a value -flag-clear autostart # unset a flag -``` - -Any project script can `source lib/flags.sh` (or the installed `/usr/local/bin/flags.sh`) and use `flag_set`, `flag_is_set`, `flag_value`, `flag_clear`. - ---- - -## The `pos` CLI - -After install, use the `pos` command for everything: - -```bash -pos # list available categories -pos help network # help for a specific category -``` - -### Network - -```bash -pos network ip # Show interfaces, routes, public IP -pos network checkport 192.168.1.1:80 # Check if a TCP port is open -pos network scan 192.168.8.0/24 # Fast parallel ping sweep -``` - -### Docker - -```bash -pos docker ps # List containers with health, IPs, ports -pos docker health # Health dashboard (exits 1 if unhealthy) -``` - -#### Compose (ScaleTail — 119+ self-hosted services) - -Each service runs with a Tailscale sidecar and gets its own `tail-xxxxx.ts.net` URL. - -**Quick start:** - -```bash -# 1. Set your Tailscale auth key (required once) -pos docker compose config set TS_AUTHKEY=tskey-auth-xxxxx - -# 2. Deploy a service -pos docker compose up jellyfin - -# 3. Open https://jellyfin.tail-xxxxx.ts.net -``` - -**All commands:** - -```bash -pos docker compose ls # List available service templates -pos docker compose installed # List deployed services -pos docker compose up jellyfin # Deploy or start a service -pos docker compose down actual-budget # Stop a service -pos docker compose logs home-assistant -f # Tail logs -pos docker compose restart home-assistant # Restart a service -pos docker compose update # Pull latest templates + refresh deployed compose files -pos docker compose config # Show current configuration -pos docker compose config set TZ=Asia/Tokyo # Set a global default -pos docker compose config edit # Open config in editor -``` - -**Config strategy — three layers:** - -| Layer | File | Purpose | -|-------|------|---------| -| Template defaults | `/usr/local/share/linux_post_install/scale-tail/services//.env` | Per-service defaults from ScaleTail | -| Global config | `~/.config/linux_post_install/compose.env` | Your defaults — applies to all services | -| Per-service | `/srv//.env` | Actual config — created on first deploy, **never overwritten** | - -Set global defaults once, then every `up` fills them into the new service's `.env`. - -**Paths:** - -- Templates: `/usr/local/share/linux_post_install/scale-tail/services/` -- Deployments: `/srv//` (configurable via `SERVICES_BASE`) -- Global config: `~/.config/linux_post_install/compose.env` - -**Config keys:** - -| Key | What it does | -|-----|-------------| -| `TS_AUTHKEY` | Tailscale auth key (required for sidecar networking) | -| `TZ` | Timezone for the service | -| `DNS_SERVER` | Custom DNS server | -| `SERVICES_BASE` | Where services are deployed (default: `/srv`) - -### VBox (disposable Docker containers) - -```bash -pos vbox create lab1 # Create — prompts to enter -pos vbox create lab1 --dir . # Bind mount current directory -pos vbox enter lab1 # Auto-starts if stopped -pos vbox ls # List vbox containers only -pos vbox stop/start/rm lab1 -``` - -### Media - -```bash -pos media mp3 https://youtube.com/watch?v=... # Audio → MP3 -pos media mp4 https://youtube.com/watch?v=... # Video with format selection -``` - -### System - -```bash -sudo pos system firewall # Interactive UFW manager -``` - -### SSH - -```bash -pos ssh load-keys # Load all SSH keys into agent -``` - -The `ssh-agent.service` runs at boot. `SSH_AUTH_SOCK` is set in `~/.bashrc`. - -### Legacy wrappers - -These still work and forward to `pos`: `wr-ip`, `wr-checkport`, `wr-scan-ping`, `wr-docker`, `wr-compose`, `wr-ufw`, `mp3`, `mp4`, `vbox`, `ssh-load-all`. - ---- - -## Optional Apps - -Install with `./apps/install.sh` (interactive), `./apps/install.sh --all`, or by name. -Uninstall the same way with `--uninstall`: - -```bash -./apps/install.sh --uninstall # interactive uninstall selection -./apps/install.sh --uninstall --all # uninstall everything -./apps/install.sh --uninstall brave vscode # uninstall specific apps -``` - -| Category | Apps | -|----------|------| -| Browsers | Brave | -| Development | opencode, VS Code | -| Media | OBS Studio, scrcpy, VLC | -| Networking | NetBird, Tailscale, ZeroTier | -| Remote Access | Termius, VNC Viewer | -| System | Docker Engine, QEMU + KVM | -| Utilities | AFFiNE, btop, LocalSend | - ---- - -## Development - -See [DEV.md](DEV.md) for architecture, conventions, and how to add or modify tools. +| Topic | Where | +|-------|-------| +| Docs index | [DOC/README.md](DOC/README.md) | +| Core scripts (installer, libs, features) — how they work + config | [DOC/SCRIPTS.md](DOC/SCRIPTS.md) | +| `pos` CLI reference (all commands, compose config, wrappers) | [DOC/POS.md](DOC/POS.md) | +| Optional apps (picker + full catalog) | [DOC/APPS.md](DOC/APPS.md) | +| Systemd services & bash completion | [DOC/SYSTEMD.md](DOC/SYSTEMD.md) | +| Developer guide (add tools/apps/features) | [DOC/DEV.md](DOC/DEV.md) | +| Algorithm diagrams | [DOC/algorithm.md](DOC/algorithm.md) |