feat: add pos system health, lib/notify.sh, and daily Telegram digest timer
- bin/pos-system-health: host dashboard (disk, RAM, failed units, backup age,
fail2ban, docker); exits 1 on any FAIL; --send/--markdown via Telegram
- lib/notify.sh: self-contained opt-in alerting helper; silent-fails, wired
into pos-system-backup (success + ERR trap) and pos-system-firewall
- systemd/pos-health.{service,timer}: 08:00 digest as installing user;
postinstall enables timer once telegram.env exists, now copies *.timer
- AGENT_TODO.md: worklist with Now/Next/Later/Done history; linked from AGENTS.md
This commit is contained in:
@@ -15,3 +15,4 @@ CRITICAL: real guidance lives in DOC/. When you encounter a reference below, use
|
|||||||
|
|
||||||
- Each tool is `bin/pos-<category>-<command>`; `bin/pos` dispatches via smart arg matching; bash completion derives from filenames.
|
- Each tool is `bin/pos-<category>-<command>`; `bin/pos` dispatches via smart arg matching; bash completion derives from filenames.
|
||||||
- Verify edits with `make check` (runs `bash -n` + the generated-doc sync gate). Generated sections (dispatch table, file table, line counts, completion flags, this doc's Document Map) are code-derived — after changing `bin/pos-*`, `lib/*`, or `completions/`, run `make gen` and commit the refreshed output.
|
- Verify edits with `make check` (runs `bash -n` + the generated-doc sync gate). Generated sections (dispatch table, file table, line counts, completion flags, this doc's Document Map) are code-derived — after changing `bin/pos-*`, `lib/*`, or `completions/`, run `make gen` and commit the refreshed output.
|
||||||
|
- Maintain `AGENT_TODO.md` (Now / Next / Later / Done): when you finish a task, move it to **Done** (dated) in the same commit.
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# AGENT_TODO — Worklist & Idea Backlog
|
||||||
|
|
||||||
|
Living list of what we are doing, what is next, and what we might do later.
|
||||||
|
Deep history lives in git: `git log --follow AGENT_TODO.md`, `git blame`, and
|
||||||
|
the individual feature commits — the **Done** section below is just a readable
|
||||||
|
summary (newest last).
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Now** — items actively being worked on this session (only a few).
|
||||||
|
- **Next** — queued, well-scoped items.
|
||||||
|
- **Later** — idea backlog. Ideas marked **NOT NOW** were evaluated and rejected
|
||||||
|
for the stated reason; revisit only if circumstances change.
|
||||||
|
- When a task is completed: move it from Now/Next into **Done** (dated one-line)
|
||||||
|
in the same commit that finishes the work.
|
||||||
|
|
||||||
|
## Now
|
||||||
|
|
||||||
|
- (none — Tier 1 shipped: `pos system health`, `lib/notify.sh`, digest timer)
|
||||||
|
|
||||||
|
## Next
|
||||||
|
|
||||||
|
- Wire alerting into more tools as they are added (default: source
|
||||||
|
`lib/notify.sh`, call `notify_send` on success/failure).
|
||||||
|
|
||||||
|
## Later — idea backlog
|
||||||
|
|
||||||
|
- **Tier 2: watch plugins** — `pos system watch <event>`: poll conditions and
|
||||||
|
alert on change (public IP changed, disk > 90%, backup skipped, fail2ban
|
||||||
|
spike). Reuses `notify_send` + a systemd timer per watch.
|
||||||
|
- **Tier 2: `pos health` extras** — temperature/fan/load average thresholds,
|
||||||
|
`ss -tln` port checks for known services, SMART status for disks.
|
||||||
|
- **Tier 3: backup rotation + remote target** — keep-N rotations, upload to
|
||||||
|
rclone remote after verify, `--remote` flag, digest reports rotation age.
|
||||||
|
- **Tier 3: `pos secret` vault** — gpg/age-encrypted key-value store; backend
|
||||||
|
for future tools that need stored tokens.
|
||||||
|
- **Tier 3: `pos inventory`** — machine manifest (OS, packages, services,
|
||||||
|
mounted disks, USB devices) exportable as markdown/JSON.
|
||||||
|
- **Tier 4: `pos self update`** — pull repo, `make gen && make check`,
|
||||||
|
re-run install.sh to refresh `/usr/local/bin`.
|
||||||
|
- **Tier 4: `pos new`** — scaffold a new tool from `templates/pos-tool.sh`
|
||||||
|
(category, name, POS header, exec bit, doc stubs).
|
||||||
|
- **NOT NOW:** Telegram inbound bot (command handling) — outbound alerting
|
||||||
|
covers current needs; revisit if remote control is wanted.
|
||||||
|
- **NOT NOW:** per-category `bin/` subdirectories — flat `bin/` + filename
|
||||||
|
dispatch scales fine; revisit only if `bin/` passes ~40 files.
|
||||||
|
- **NOT NOW:** split `lib/entertainment-lib.sh` — fine under 600 lines; revisit
|
||||||
|
if it grows.
|
||||||
|
|
||||||
|
## Done (summary, newest last)
|
||||||
|
|
||||||
|
- 2026-08-06: Tier 1 — `pos system health` (dashboard + `--send`), `lib/notify.sh`
|
||||||
|
(wired into backup + firewall), daily digest timer via postinstall.
|
||||||
|
- 2026-08-06: Document Map index + Entertainment section in AGENT_Context (cf36780).
|
||||||
|
- 2026-08-06: Entertainment module — plugins (weather/joke/gold), `pos
|
||||||
|
entertainment config/enable/disable/send/status`, auto-trigger + Telegram send.
|
||||||
|
- 2026-08-05: `pos communication telegram` — `--parse-mode` (plain/markdown/html).
|
||||||
|
- 2026-08-05: doc/code sync gate — `make gen` + `make check` + pre-commit hook.
|
||||||
|
- 2026-08-05: `pos usb server` — USB Redirector control tool (494eae2).
|
||||||
|
- 2026-08-05: `pos <category> --help` auto-discovery in the dispatcher.
|
||||||
|
- 2026-08-05: AGENTS.md with lazy-loaded DOC references.
|
||||||
@@ -10,19 +10,19 @@
|
|||||||
|
|
||||||
<!-- GEN:START docmap -->
|
<!-- GEN:START docmap -->
|
||||||
| ## 1. Project Overview | 28–43 |
|
| ## 1. Project Overview | 28–43 |
|
||||||
| ## 2. Directory Structure | 44–164 |
|
| ## 2. Directory Structure | 44–165 |
|
||||||
| ## 3. Installation Flow | 165–216 |
|
| ## 3. Installation Flow | 166–217 |
|
||||||
| ## 4. The `pos` CLI System | 217–273 |
|
| ## 4. The `pos` CLI System | 218–275 |
|
||||||
| ## 5. Shared Library — `lib/common.sh` | 274–304 |
|
| ## 5. Shared Library — `lib/common.sh` | 276–306 |
|
||||||
| ## 6. Docker Compose / ScaleTail | 305–347 |
|
| ## 6. Docker Compose / ScaleTail | 307–349 |
|
||||||
| ## 7. Optional Apps (`apps/`) | 348–377 |
|
| ## 7. Optional Apps (`apps/`) | 350–379 |
|
||||||
| ## 8. Entertainment Module | 378–391 |
|
| ## 8. Entertainment Module | 380–393 |
|
||||||
| ## 9. Systemd Services | 392–402 |
|
| ## 9. Systemd Services | 394–404 |
|
||||||
| ## 10. Configuration Files | 403–426 |
|
| ## 10. Configuration Files | 405–428 |
|
||||||
| ## 11. Coding Conventions | 427–459 |
|
| ## 11. Coding Conventions | 429–461 |
|
||||||
| ## 12. Development Workflow | 460–511 |
|
| ## 12. Development Workflow | 462–513 |
|
||||||
| ## 13. Key File Quick Reference | 512–553 |
|
| ## 13. Key File Quick Reference | 514–556 |
|
||||||
| ## 14. Common Tasks for Agents | 554–578 |
|
| ## 14. Common Tasks for Agents | 557–581 |
|
||||||
<!-- GEN:END docmap -->
|
<!-- GEN:END docmap -->
|
||||||
|
|
||||||
## 1. Project Overview
|
## 1. Project Overview
|
||||||
@@ -76,6 +76,7 @@ Linux_post_install/
|
|||||||
│ ├── pos-ssh-load-keys # Load all SSH keys into the agent
|
│ ├── pos-ssh-load-keys # Load all SSH keys into the agent
|
||||||
│ ├── pos-system-backup # Encrypted (AES-256) folder snapshots (tar + gpg)
|
│ ├── pos-system-backup # Encrypted (AES-256) folder snapshots (tar + gpg)
|
||||||
│ ├── pos-system-firewall # Interactive UFW management
|
│ ├── pos-system-firewall # Interactive UFW management
|
||||||
|
│ ├── pos-system-health # Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL
|
||||||
│ ├── pos-usb-server # USB Redirector server control (--ls, --share; prompts when args omitted)
|
│ ├── pos-usb-server # USB Redirector server control (--ls, --share; prompts when args omitted)
|
||||||
<!-- GEN:END tree -->
|
<!-- GEN:END tree -->
|
||||||
│ ├── flag-reader # Inspect feature flags (list/status/--raw)
|
│ ├── flag-reader # Inspect feature flags (list/status/--raw)
|
||||||
@@ -252,6 +253,7 @@ All non-interactive `pos` commands log output to `~/.local/share/linux_post_inst
|
|||||||
| ssh | load-keys | `pos-ssh-load-keys` | Load all SSH keys into the agent |
|
| ssh | load-keys | `pos-ssh-load-keys` | Load all SSH keys into the agent |
|
||||||
| system | backup | `pos-system-backup` | Encrypted (AES-256) folder snapshots (tar + gpg) |
|
| system | backup | `pos-system-backup` | Encrypted (AES-256) folder snapshots (tar + gpg) |
|
||||||
| system | firewall | `pos-system-firewall` | Interactive UFW management |
|
| system | firewall | `pos-system-firewall` | Interactive UFW management |
|
||||||
|
| system | health | `pos-system-health` | Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL |
|
||||||
| usb | server | `pos-usb-server` | USB Redirector server control (--ls, --share; prompts when args omitted) |
|
| usb | server | `pos-usb-server` | USB Redirector server control (--ls, --share; prompts when args omitted) |
|
||||||
<!-- GEN:END dispatch -->
|
<!-- GEN:END dispatch -->
|
||||||
|
|
||||||
@@ -542,10 +544,11 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
|||||||
| `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location |
|
| `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location |
|
||||||
| `bin/pos-network-scan` | 271 | Parallel ping sweep of CIDR |
|
| `bin/pos-network-scan` | 271 | Parallel ping sweep of CIDR |
|
||||||
| `bin/pos-ssh-load-keys` | 31 | Load all SSH keys into the agent |
|
| `bin/pos-ssh-load-keys` | 31 | Load all SSH keys into the agent |
|
||||||
| `bin/pos-system-backup` | 117 | Encrypted (AES-256) folder snapshots (tar + gpg) |
|
| `bin/pos-system-backup` | 121 | Encrypted (AES-256) folder snapshots (tar + gpg) |
|
||||||
| `bin/pos-system-firewall` | 285 | Interactive UFW management |
|
| `bin/pos-system-firewall` | 291 | Interactive UFW management |
|
||||||
|
| `bin/pos-system-health` | 230 | Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL |
|
||||||
| `bin/pos-usb-server` | 218 | USB Redirector server control (--ls, --share; prompts when args omitted) |
|
| `bin/pos-usb-server` | 218 | USB Redirector server control (--ls, --share; prompts when args omitted) |
|
||||||
| `completions/pos.bash` | 188 | Dynamic bash completion |
|
| `completions/pos.bash` | 189 | Dynamic bash completion |
|
||||||
<!-- GEN:END filetable -->
|
<!-- GEN:END filetable -->
|
||||||
| `apps/install.sh` | 171 | App install/uninstall picker/orchestrator |
|
| `apps/install.sh` | 171 | App install/uninstall picker/orchestrator |
|
||||||
|
|
||||||
|
|||||||
+12
@@ -275,6 +275,18 @@ Place it in `apps/<category>/<name>.sh`. It auto-appears in the picker — no re
|
|||||||
|
|
||||||
## Best Practices
|
## Best Practices
|
||||||
|
|
||||||
|
### Alerting
|
||||||
|
|
||||||
|
To notify on events (Telegram), source the shared helper instead of calling the telegram tool directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
|
||||||
|
notify_send "Backup completed"
|
||||||
|
notify_send "**disk full**" --markdown
|
||||||
|
```
|
||||||
|
|
||||||
|
`notify_send` is deliberately dependency-free (defines only itself, so it never clobbers a tool's own `log`/`warn`/`err`) and **silent-fails**: if Telegram is missing or not configured it warns and returns 0, never breaking the caller's flow or exit code. Source it opt-in in any tool that should alert; for failure alerts use `trap 'notify_send "..." ERR'`.
|
||||||
|
|
||||||
### Idempotency
|
### Idempotency
|
||||||
|
|
||||||
Check before creating, use `>>` with grep guards, don't overwrite user configs.
|
Check before creating, use `>>` with grep guards, don't overwrite user configs.
|
||||||
|
|||||||
+5
-2
@@ -151,9 +151,12 @@ The standalone `vbox` command still works and forwards to `pos docker vbox` (see
|
|||||||
|
|
||||||
| Command | File | Purpose | Configuration |
|
| Command | File | Purpose | Configuration |
|
||||||
|---------|------|---------|---------------|
|
|---------|------|---------|---------------|
|
||||||
| `sudo pos system firewall` | `bin/pos-system-firewall` | Interactive UFW ("UFW POWER") menu: add/delete rules, status, enable/disable/reset, default policies | Must run as root. Every command is previewed and confirmed before execution; supports `--dry-run`; keeps a history of executed commands |
|
| `sudo pos system firewall` | `bin/pos-system-firewall` | Interactive UFW ("UFW POWER") menu: add/delete rules, status, enable/disable/reset, default policies | Must run as root. Every command is previewed and confirmed before execution; supports `--dry-run`; keeps a history of executed commands. Executed mutating changes are announced via `lib/notify.sh` |
|
||||||
| `pos system backup <folder-path>` | `bin/pos-system-backup` | Create a gpg-encrypted (AES-256) `tar.gz` snapshot of a folder and verify it | Prompts twice for a password (never stored). Uses `sudo tar`; needs `gnupg` (in `preinstall.sh` PACKAGES). Artifact `<name>_<date>.tar.gz.gpg` in the current directory, `chmod 600` |
|
| `pos system backup <folder-path>` | `bin/pos-system-backup` | Create a gpg-encrypted (AES-256) `tar.gz` snapshot of a folder and verify it | Prompts twice for a password (never stored). Uses `sudo tar`; needs `gnupg` (in `preinstall.sh` PACKAGES). Artifact `<name>_<date>.tar.gz.gpg` in the current directory, `chmod 600`. Success/failure are announced via `lib/notify.sh` |
|
||||||
| `pos system backup --service` | `bin/pos-system-backup` | Lists folders under `/srv` and `~/srv`, lets you pick one, then runs the same backup | Roots via `BACKUP_SERVICE_ROOTS` (space-separated, default `/srv $HOME/srv`) |
|
| `pos system backup --service` | `bin/pos-system-backup` | Lists folders under `/srv` and `~/srv`, lets you pick one, then runs the same backup | Roots via `BACKUP_SERVICE_ROOTS` (space-separated, default `/srv $HOME/srv`) |
|
||||||
|
| `pos system health [--send] [--markdown]` | `bin/pos-system-health` | Host health dashboard: disk per mount, RAM/swap, failed systemd units, backup age, fail2ban, docker containers. Exits 1 if any check FAILs | `--send`/`--markdown` send the summary via Telegram (`lib/notify.sh`). Backup age threshold via `HEALTH_BACKUP_MAX_AGE_DAYS` (default 2); backup search roots via `BACKUP_SERVICE_ROOTS` |
|
||||||
|
|
||||||
|
`systemd/pos-health.service` + `systemd/pos-health.timer` run `pos system health --send --markdown` daily at 08:00 as the installing user. `postinstall.sh` enables the timer automatically once `~/.config/linux_post_install/telegram.env` exists — re-run postinstall after configuring Telegram to pick it up.
|
||||||
|
|
||||||
### ssh
|
### ssh
|
||||||
|
|
||||||
|
|||||||
+32
-3
@@ -5,6 +5,7 @@ The units installed and enabled by `postinstall.sh`, plus the `pos` bash complet
|
|||||||
- [Services](#services)
|
- [Services](#services)
|
||||||
- [`autostart.service`](#autostartservice)
|
- [`autostart.service`](#autostartservice)
|
||||||
- [`ssh-agent.service`](#ssh-agentservice)
|
- [`ssh-agent.service`](#ssh-agentservice)
|
||||||
|
- [`pos-health.service`](#pos-healthservice)
|
||||||
- [Feature-flag gating](#feature-flag-gating)
|
- [Feature-flag gating](#feature-flag-gating)
|
||||||
- [Bash completion](#bash-completion)
|
- [Bash completion](#bash-completion)
|
||||||
|
|
||||||
@@ -12,7 +13,7 @@ The units installed and enabled by `postinstall.sh`, plus the `pos` bash complet
|
|||||||
|
|
||||||
## Services
|
## Services
|
||||||
|
|
||||||
`postinstall.sh` copies every `systemd/*.service` to `/etc/systemd/system/`, runs `systemctl daemon-reload`, then enables each one (see the gating rule below).
|
`postinstall.sh` copies every `systemd/*.service` (and `systemd/*.timer`) to `/etc/systemd/system/`, runs `systemctl daemon-reload`, then enables each one (see the gating rule below).
|
||||||
|
|
||||||
### autostart.service
|
### autostart.service
|
||||||
|
|
||||||
@@ -59,20 +60,48 @@ WantedBy=multi-user.target
|
|||||||
|
|
||||||
**Configuration:** socket at `/run/ssh-agent/socket` (world-readable/writable). `~/.bashrc` (set by `postinstall.sh`) exports `SSH_AUTH_SOCK` to it. Not gated on any feature flag.
|
**Configuration:** socket at `/run/ssh-agent/socket` (world-readable/writable). `~/.bashrc` (set by `postinstall.sh`) exports `SSH_AUTH_SOCK` to it. Not gated on any feature flag.
|
||||||
|
|
||||||
|
### pos-health.service
|
||||||
|
|
||||||
|
**Purpose:** daily "health digest" — runs `pos system health --send --markdown` at 08:00 and sends the report to Telegram.
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=POS Health digest (daily report via Telegram)
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=__POS_USER__
|
||||||
|
ExecStart=/usr/local/bin/pos system health --send --markdown
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 08:00:00
|
||||||
|
Persistent=true
|
||||||
|
```
|
||||||
|
|
||||||
|
The service is `Type=oneshot` and is driven **only** by its companion `pos-health.timer` (`WantedBy=timers.target`); the service itself is never enabled directly.
|
||||||
|
|
||||||
|
**Configuration:** `postinstall.sh` substitutes `__POS_USER__` with the installing user (`${SUDO_USER:-$USER}`) so the digest uses that user's real Telegram config. The timer is enabled only when `~/.config/linux_post_install/telegram.env` already exists — otherwise postinstall warns and skips; re-run postinstall after configuring Telegram (`pos communication telegram config set TELEGRAM_*`) to install it.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Feature-flag gating
|
## Feature-flag gating
|
||||||
|
|
||||||
The systemd loop in `postinstall.sh` special-cases `autostart.service`:
|
The systemd loop in `postinstall.sh` special-cases two units:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
if [ "$svc_name" = "autostart.service" ] && ! flag_is_set autostart; then
|
if [ "$svc_name" = "autostart.service" ] && ! flag_is_set autostart; then
|
||||||
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
|
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
if [ "$svc_name" = "pos-health.service" ]; then
|
||||||
|
# substitute User= and enable pos-health.timer only if Telegram is configured
|
||||||
|
fi
|
||||||
```
|
```
|
||||||
|
|
||||||
Set the flag with `./install.sh --feature` (or `flag-set autostart`). See [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags).
|
- `autostart.service` is **enabled** only when the `autostart` feature flag is set (`./install.sh --feature` or `flag-set autostart`). See [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags).
|
||||||
|
- `pos-health.service` is **not** enabled at all — `postinstall.sh` enables `pos-health.timer` instead, and only when a Telegram config already exists.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ It is a **personal toolkit** — a bootstrap script, a unified `pos` CLI for eve
|
|||||||
- Wi-Fi hotspot tools (`create_ap`, `wihotspot-gui`) via `pos network hotspot`
|
- Wi-Fi hotspot tools (`create_ap`, `wihotspot-gui`) via `pos network hotspot`
|
||||||
- 15 optional desktop apps (VS Code, Brave, OBS, Tailscale, …) — pick what you want
|
- 15 optional desktop apps (VS Code, Brave, OBS, Tailscale, …) — pick what you want
|
||||||
- 119+ self-hosted services with Tailscale access (Jellyfin, Home Assistant, …)
|
- 119+ self-hosted services with Tailscale access (Jellyfin, Home Assistant, …)
|
||||||
- systemd services for SSH agent and boot-time automation
|
- systemd services for SSH agent, boot-time automation, and a daily Telegram health digest (`pos system health`)
|
||||||
- Everything lands in `/usr/local/bin/` — you can delete the repo after install
|
- Everything lands in `/usr/local/bin/` — you can delete the repo after install
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|||||||
@@ -4,6 +4,9 @@ set -euo pipefail
|
|||||||
# POS_FLAGS: --service
|
# POS_FLAGS: --service
|
||||||
|
|
||||||
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
|
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
|
||||||
|
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
|
||||||
|
|
||||||
|
trap 'notify_send "Backup FAILED: ${FOLDER:-unknown}"' ERR
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
cat <<EOF
|
cat <<EOF
|
||||||
@@ -115,3 +118,4 @@ gpg --batch --quiet --passphrase "$PASS" --decrypt "$ARCHIVE" | tar -tzf - > /de
|
|||||||
unset PASS
|
unset PASS
|
||||||
echo
|
echo
|
||||||
log "Backup completed: $ARCHIVE"
|
log "Backup completed: $ARCHIVE"
|
||||||
|
notify_send "Backup completed: $ARCHIVE"
|
||||||
|
|||||||
@@ -9,6 +9,8 @@ if [[ $EUID -ne 0 ]]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
|
||||||
|
|
||||||
HISTORY=()
|
HISTORY=()
|
||||||
DRY_RUN=0
|
DRY_RUN=0
|
||||||
|
|
||||||
@@ -29,6 +31,10 @@ run_cmd() {
|
|||||||
echo "(dry-run) skipping execution"
|
echo "(dry-run) skipping execution"
|
||||||
else
|
else
|
||||||
"${cmd[@]}"
|
"${cmd[@]}"
|
||||||
|
# Alert on mutating commands only (status queries are read-only)
|
||||||
|
if [[ "${cmd[1]:-}" != "status" ]]; then
|
||||||
|
notify_send "Firewall: ${cmd[*]}"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
HISTORY+=("${cmd[*]}")
|
HISTORY+=("${cmd[*]}")
|
||||||
else
|
else
|
||||||
|
|||||||
Executable
+230
@@ -0,0 +1,230 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
# POS: system health — Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL
|
||||||
|
# POS_FLAGS: --send --markdown
|
||||||
|
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
|
||||||
|
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
|
||||||
|
|
||||||
|
SEND=0
|
||||||
|
MARKDOWN=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
-h|--help) usage_placeholder=1 ;;
|
||||||
|
--send) SEND=1 ;;
|
||||||
|
--markdown) MARKDOWN=1 ;;
|
||||||
|
*) err "Unknown option '$arg' (see --help)" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "${usage_placeholder:-0}" -eq 1 ]; then
|
||||||
|
cat <<EOF
|
||||||
|
Usage: pos system health [--send] [--markdown]
|
||||||
|
|
||||||
|
Host health dashboard: disk, RAM/swap, failed systemd units, backup age,
|
||||||
|
fail2ban, docker containers. Exits 1 if any check FAILs.
|
||||||
|
|
||||||
|
Flags:
|
||||||
|
--send Send the summary via Telegram (uses lib/notify.sh)
|
||||||
|
--markdown Same as --send, with markdown parse mode
|
||||||
|
-h, --help Show this help
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
HEALTH_BACKUP_MAX_AGE_DAYS Max backup age before a WARN (default: 2)
|
||||||
|
BACKUP_SERVICE_ROOTS Where to look for backups (default: /srv \$HOME/srv)
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
pos system health
|
||||||
|
pos system health --send
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$MARKDOWN" -eq 1 ]; then
|
||||||
|
SEND=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
FAILURES=0
|
||||||
|
WARNINGS=0
|
||||||
|
REPORT=()
|
||||||
|
|
||||||
|
report() {
|
||||||
|
local st="$1" label="$2" detail="$3" line
|
||||||
|
case "$st" in
|
||||||
|
ok)
|
||||||
|
echo " ${GREEN}${BOLD}[ OK ]${RESET} ${label}: ${detail}"
|
||||||
|
printf -v line "[ OK ] %s: %s" "$label" "$detail"
|
||||||
|
;;
|
||||||
|
warn)
|
||||||
|
echo " ${YELLOW}${BOLD}[WARN]${RESET} ${label}: ${detail}"
|
||||||
|
printf -v line "[WARN] %s: %s" "$label" "$detail"
|
||||||
|
WARNINGS=$((WARNINGS + 1))
|
||||||
|
;;
|
||||||
|
fail)
|
||||||
|
echo " ${RED}${BOLD}[FAIL]${RESET} ${label}: ${detail}"
|
||||||
|
printf -v line "[FAIL] %s: %s" "$label" "$detail"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
REPORT+=("$line")
|
||||||
|
}
|
||||||
|
|
||||||
|
host="$(hostname -s 2>/dev/null || echo "localhost")"
|
||||||
|
ip_public="$(curl -fsS -m 5 https://api.ipify.org 2>/dev/null || echo "unreachable")"
|
||||||
|
loadavg="$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo "?")"
|
||||||
|
uptime_s="$(uptime -p 2>/dev/null || echo "?")"
|
||||||
|
|
||||||
|
section "System Health — ${host}"
|
||||||
|
echo " ${BLUE}${BOLD}date${RESET} $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
echo " ${BLUE}${BOLD}uptime${RESET} ${uptime_s}"
|
||||||
|
echo " ${BLUE}${BOLD}load${RESET} ${loadavg}"
|
||||||
|
echo " ${BLUE}${BOLD}ip${RESET} ${ip_public}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# ── Disk ──────────────────────────────────────────────────────
|
||||||
|
DISK_PCT_MAX=0
|
||||||
|
DISK_ISSUES=()
|
||||||
|
while read -r fs use mount; do
|
||||||
|
pct=${use%\%}
|
||||||
|
if [ "$pct" -gt "$DISK_PCT_MAX" ]; then
|
||||||
|
DISK_PCT_MAX="$pct"
|
||||||
|
fi
|
||||||
|
if [ "$pct" -ge 90 ]; then
|
||||||
|
DISK_ISSUES+=("${mount} ${pct}%")
|
||||||
|
elif [ "$pct" -ge 85 ]; then
|
||||||
|
DISK_ISSUES+=("${mount} ${pct}%")
|
||||||
|
fi
|
||||||
|
done < <(df -P -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs -x proc -x sysfs -x cgroup2 -x zfs -x aufs 2>/dev/null | awk 'NR>1 && $1 !~ /loop|sr[0-9]/ {print $1, $5, $6}')
|
||||||
|
|
||||||
|
if [ ${#DISK_ISSUES[@]} -gt 0 ]; then
|
||||||
|
report fail "disk" "${DISK_ISSUES[*]}"
|
||||||
|
else
|
||||||
|
report ok "disk" "max usage ${DISK_PCT_MAX}%"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── RAM / swap ────────────────────────────────────────────────
|
||||||
|
mem=($(free -m | awk '/^Mem:/{print $2, $7}'))
|
||||||
|
swap=($(free -m | awk '/^Swap:/{print $2, $3}'))
|
||||||
|
MEM_TOTAL=${mem[0]:-0}
|
||||||
|
MEM_AVAIL=${mem[1]:-0}
|
||||||
|
MEM_PCT=100
|
||||||
|
if [ "$MEM_TOTAL" -gt 0 ]; then
|
||||||
|
MEM_PCT=$(( (MEM_TOTAL - MEM_AVAIL) * 100 / MEM_TOTAL ))
|
||||||
|
fi
|
||||||
|
SWAP_PCT=0
|
||||||
|
if [ "${swap[0]:-0}" -gt 0 ]; then
|
||||||
|
SWAP_PCT=$(( swap[1] * 100 / swap[0] ))
|
||||||
|
fi
|
||||||
|
if [ "$MEM_PCT" -ge 95 ] || [ "$SWAP_PCT" -ge 90 ]; then
|
||||||
|
report fail "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
|
||||||
|
elif [ "$MEM_PCT" -ge 85 ]; then
|
||||||
|
report warn "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
|
||||||
|
else
|
||||||
|
report ok "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Failed systemd units ──────────────────────────────────────
|
||||||
|
if command -v systemctl >/dev/null 2>&1; then
|
||||||
|
failed_units=()
|
||||||
|
while IFS= read -r u; do
|
||||||
|
[ -n "$u" ] && failed_units+=("${u%% *}")
|
||||||
|
done < <(systemctl --failed --plain --no-legend --no-pager 2>/dev/null || true)
|
||||||
|
if [ "${#failed_units[@]}" -gt 0 ]; then
|
||||||
|
report fail "services" "${#failed_units[@]} failed unit(s): ${failed_units[*]}"
|
||||||
|
else
|
||||||
|
report ok "services" "no failed units"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
report warn "services" "systemd not available"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Backup age ────────────────────────────────────────────────
|
||||||
|
MAX_AGE="${HEALTH_BACKUP_MAX_AGE_DAYS:-2}"
|
||||||
|
roots=()
|
||||||
|
if [ -n "${BACKUP_SERVICE_ROOTS:-}" ]; then
|
||||||
|
read -r -a roots <<< "$BACKUP_SERVICE_ROOTS"
|
||||||
|
else
|
||||||
|
roots=(/srv "$HOME/srv")
|
||||||
|
fi
|
||||||
|
newest=""
|
||||||
|
newest_ts=0
|
||||||
|
for root in "${roots[@]}"; do
|
||||||
|
[ -d "$root" ] || continue
|
||||||
|
while IFS= read -r line; do
|
||||||
|
ts="${line%% *}"
|
||||||
|
if [ "${ts%%.*}" -gt "$newest_ts" ]; then
|
||||||
|
newest_ts="${ts%%.*}"
|
||||||
|
newest="${line#* }"
|
||||||
|
fi
|
||||||
|
done < <(find "$root" -maxdepth 2 -type f -name '*.tar.gz.gpg' -printf '%T@ %p\n' 2>/dev/null || true)
|
||||||
|
done
|
||||||
|
if [ -z "$newest" ]; then
|
||||||
|
report warn "backup" "no backups found under ${roots[*]}"
|
||||||
|
else
|
||||||
|
now="$(date +%s)"
|
||||||
|
days=$(( (now - newest_ts) / 86400 ))
|
||||||
|
if [ "$days" -gt "$MAX_AGE" ]; then
|
||||||
|
report warn "backup" "${days}d old: ${newest}"
|
||||||
|
else
|
||||||
|
report ok "backup" "${days}d old: ${newest}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── fail2ban ──────────────────────────────────────────────────
|
||||||
|
if command -v fail2ban-client >/dev/null 2>&1; then
|
||||||
|
if fail2ban-client ping >/dev/null 2>&1; then
|
||||||
|
banned=0
|
||||||
|
jails=$(fail2ban-client status 2>/dev/null | awk -F': *' '/Jail list/{print $2}')
|
||||||
|
for j in $jails; do
|
||||||
|
n=$(fail2ban-client status "$j" 2>/dev/null | awk -F': *' '/Total banned/{print $2}')
|
||||||
|
n="${n:-0}"
|
||||||
|
banned=$(( banned + (n > 0 ? n : 0) ))
|
||||||
|
done
|
||||||
|
report ok "fail2ban" "running, ${banned} total banned across ${jails:-0} jail(s)"
|
||||||
|
else
|
||||||
|
report warn "fail2ban" "installed but not running"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Docker ────────────────────────────────────────────────────
|
||||||
|
if command -v docker >/dev/null 2>&1; then
|
||||||
|
bad_containers=()
|
||||||
|
running=0
|
||||||
|
while IFS= read -r name state; do
|
||||||
|
case "$state" in
|
||||||
|
restarting|unhealthy) bad_containers+=("${name}(${state})") ;;
|
||||||
|
esac
|
||||||
|
done < <(docker ps -a --format '{{.Names}} {{.State}}' 2>/dev/null || true)
|
||||||
|
if [ "${#bad_containers[@]}" -gt 0 ]; then
|
||||||
|
report fail "docker" "${#bad_containers[@]} problem container(s): ${bad_containers[*]}"
|
||||||
|
else
|
||||||
|
report ok "docker" "no unhealthy/restarting containers"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
if [ "$FAILURES" -gt 0 ]; then
|
||||||
|
verdict="${FAILURES} FAIL, ${WARNINGS} WARN"
|
||||||
|
echo " ${RED}${BOLD}VERDICT${RESET} ${verdict}"
|
||||||
|
elif [ "$WARNINGS" -gt 0 ]; then
|
||||||
|
verdict="${WARNINGS} WARN — mostly healthy"
|
||||||
|
echo " ${YELLOW}${BOLD}VERDICT${RESET} ${verdict}"
|
||||||
|
else
|
||||||
|
verdict="all good"
|
||||||
|
echo " ${GREEN}${BOLD}VERDICT${RESET} ${verdict}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$SEND" -eq 1 ]; then
|
||||||
|
msg="POS Health — ${host}
|
||||||
|
Verdict: ${verdict}"
|
||||||
|
for line in "${REPORT[@]}"; do
|
||||||
|
msg+="
|
||||||
|
${line}"
|
||||||
|
done
|
||||||
|
if [ "$MARKDOWN" -eq 1 ]; then
|
||||||
|
notify_send "$msg" --markdown
|
||||||
|
else
|
||||||
|
notify_send "$msg"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ "$FAILURES" -eq 0 ]
|
||||||
@@ -7,6 +7,7 @@ _pos_flags[communication-telegram]="--send --type --caption --parse-mode --no-pr
|
|||||||
_pos_flags[entertainment-send]="--print --markdown"
|
_pos_flags[entertainment-send]="--print --markdown"
|
||||||
_pos_flags[network-hotspot]="--foreground"
|
_pos_flags[network-hotspot]="--foreground"
|
||||||
_pos_flags[system-backup]="--service"
|
_pos_flags[system-backup]="--service"
|
||||||
|
_pos_flags[system-health]="--send --markdown"
|
||||||
_pos_flags[usb-server]="--ls --ls-shared --share --unshare --auto-share --callback --close-callback --auto-connect --disconnect --nickname --timeout --port --info --version"
|
_pos_flags[usb-server]="--ls --ls-shared --share --unshare --auto-share --callback --close-callback --auto-connect --disconnect --nickname --timeout --port --info --version"
|
||||||
# GEN:END posflags
|
# GEN:END posflags
|
||||||
|
|
||||||
|
|||||||
@@ -113,6 +113,7 @@ if should_run 2 scripts; then
|
|||||||
done
|
done
|
||||||
run sudo install -m 644 lib/common.sh /usr/local/bin/common.sh
|
run sudo install -m 644 lib/common.sh /usr/local/bin/common.sh
|
||||||
run sudo install -m 644 lib/flags.sh /usr/local/bin/flags.sh
|
run sudo install -m 644 lib/flags.sh /usr/local/bin/flags.sh
|
||||||
|
run sudo install -m 644 lib/notify.sh /usr/local/bin/notify.sh
|
||||||
run sudo install -m 644 lib/entertainment-lib.sh /usr/local/bin/entertainment-lib.sh
|
run sudo install -m 644 lib/entertainment-lib.sh /usr/local/bin/entertainment-lib.sh
|
||||||
|
|
||||||
# ── Entertainment plugins ────────────────────────────────
|
# ── Entertainment plugins ────────────────────────────────
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# lib/notify.sh — optional alerting helper. Self-contained by design:
|
||||||
|
# defines ONLY notify_send() so it can be sourced by tools that define
|
||||||
|
# their own log/warn/err (e.g. pos-system-firewall) without clobbering.
|
||||||
|
#
|
||||||
|
# Usage (opt-in — source it, do NOT auto-load from common.sh):
|
||||||
|
# source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
|
||||||
|
# notify_send "Backup completed: $ARCHIVE"
|
||||||
|
# notify_send "⚠ disk full" --markdown
|
||||||
|
#
|
||||||
|
# Delegates to `pos communication telegram send`; silent-fail if the
|
||||||
|
# telegram sender is missing or not configured (warns, never breaks the
|
||||||
|
# caller and never changes its exit code).
|
||||||
|
|
||||||
|
notify_send() {
|
||||||
|
local msg="" markdown=0
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--markdown) markdown=1; shift ;;
|
||||||
|
*) msg="$1"; shift ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$msg" ]; then
|
||||||
|
warn "notify_send: empty message, skipped" 2>/dev/null || echo "notify_send: empty message, skipped"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local tg
|
||||||
|
tg="$(command -v pos-communication-telegram 2>/dev/null)" || \
|
||||||
|
tg="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../bin/pos-communication-telegram"
|
||||||
|
|
||||||
|
if [ ! -x "$tg" ]; then
|
||||||
|
warn "notify_send: pos-communication-telegram not found, notification skipped" 2>/dev/null || true
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$markdown" -eq 1 ]; then
|
||||||
|
"$tg" send "$msg" --parse-mode markdown >/dev/null 2>&1 || {
|
||||||
|
warn "notify_send: telegram send failed, notification skipped" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
else
|
||||||
|
"$tg" send "$msg" >/dev/null 2>&1 || {
|
||||||
|
warn "notify_send: telegram send failed, notification skipped" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
}
|
||||||
@@ -94,8 +94,18 @@ fi
|
|||||||
# ── systemd services ───────────────────────────────────────────
|
# ── systemd services ───────────────────────────────────────────
|
||||||
if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then
|
if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then
|
||||||
run sudo cp systemd/*.service /etc/systemd/system/
|
run sudo cp systemd/*.service /etc/systemd/system/
|
||||||
|
if [ -n "$(ls -A systemd/*.timer 2>/dev/null)" ]; then
|
||||||
|
run sudo cp systemd/*.timer /etc/systemd/system/
|
||||||
|
fi
|
||||||
run sudo systemctl daemon-reload
|
run sudo systemctl daemon-reload
|
||||||
|
|
||||||
|
# pos-health.service runs as the installing user so the digest uses that
|
||||||
|
# user's real Telegram config. The timer is enabled only when a telegram.env
|
||||||
|
# is configured — re-run postinstall after configuring Telegram to pick it up.
|
||||||
|
DIGEST_USER="${SUDO_USER:-$USER}"
|
||||||
|
DIGEST_HOME="$(getent passwd "$DIGEST_USER" 2>/dev/null | cut -d: -f6)"
|
||||||
|
DIGEST_HOME="${DIGEST_HOME:-$HOME}"
|
||||||
|
|
||||||
for svc in systemd/*.service; do
|
for svc in systemd/*.service; do
|
||||||
svc_name=$(basename "$svc")
|
svc_name=$(basename "$svc")
|
||||||
# autostart.service runs features/autostart.sh — enable only when
|
# autostart.service runs features/autostart.sh — enable only when
|
||||||
@@ -104,6 +114,18 @@ if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then
|
|||||||
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
|
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
if [ "$svc_name" = "pos-health.service" ]; then
|
||||||
|
run sudo sed -i "s|__POS_USER__|$DIGEST_USER|" "/etc/systemd/system/$svc_name"
|
||||||
|
run sudo systemctl daemon-reload
|
||||||
|
if [ -f "$DIGEST_HOME/.config/linux_post_install/telegram.env" ]; then
|
||||||
|
run sudo systemctl enable --now pos-health.timer 2>/dev/null || \
|
||||||
|
run sudo systemctl enable pos-health.timer
|
||||||
|
log "Daily health digest timer enabled for $DIGEST_USER"
|
||||||
|
else
|
||||||
|
warn "Telegram not configured — skipping health digest timer (run 'pos communication telegram config set ...' then re-run postinstall)"
|
||||||
|
fi
|
||||||
|
continue
|
||||||
|
fi
|
||||||
run sudo systemctl enable --now "$svc_name" 2>/dev/null || \
|
run sudo systemctl enable --now "$svc_name" 2>/dev/null || \
|
||||||
run sudo systemctl enable "$svc_name"
|
run sudo systemctl enable "$svc_name"
|
||||||
done
|
done
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=POS Health digest (daily report via Telegram)
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=__POS_USER__
|
||||||
|
ExecStart=/usr/local/bin/pos system health --send --markdown
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Daily POS health digest timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 08:00:00
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
Reference in New Issue
Block a user