feat: add pos system health, lib/notify.sh, and daily Telegram digest timer

- bin/pos-system-health: host dashboard (disk, RAM, failed units, backup age,
  fail2ban, docker); exits 1 on any FAIL; --send/--markdown via Telegram
- lib/notify.sh: self-contained opt-in alerting helper; silent-fails, wired
  into pos-system-backup (success + ERR trap) and pos-system-firewall
- systemd/pos-health.{service,timer}: 08:00 digest as installing user;
  postinstall enables timer once telegram.env exists, now copies *.timer
- AGENT_TODO.md: worklist with Now/Next/Later/Done history; linked from AGENTS.md
This commit is contained in:
Your Name
2026-08-06 02:25:23 -04:00
parent cf36780c0f
commit 9a94329dd0
16 changed files with 460 additions and 22 deletions
+1
View File
@@ -15,3 +15,4 @@ CRITICAL: real guidance lives in DOC/. When you encounter a reference below, use
- Each tool is `bin/pos-<category>-<command>`; `bin/pos` dispatches via smart arg matching; bash completion derives from filenames. - Each tool is `bin/pos-<category>-<command>`; `bin/pos` dispatches via smart arg matching; bash completion derives from filenames.
- Verify edits with `make check` (runs `bash -n` + the generated-doc sync gate). Generated sections (dispatch table, file table, line counts, completion flags, this doc's Document Map) are code-derived — after changing `bin/pos-*`, `lib/*`, or `completions/`, run `make gen` and commit the refreshed output. - Verify edits with `make check` (runs `bash -n` + the generated-doc sync gate). Generated sections (dispatch table, file table, line counts, completion flags, this doc's Document Map) are code-derived — after changing `bin/pos-*`, `lib/*`, or `completions/`, run `make gen` and commit the refreshed output.
- Maintain `AGENT_TODO.md` (Now / Next / Later / Done): when you finish a task, move it to **Done** (dated) in the same commit.
+61
View File
@@ -0,0 +1,61 @@
# AGENT_TODO — Worklist & Idea Backlog
Living list of what we are doing, what is next, and what we might do later.
Deep history lives in git: `git log --follow AGENT_TODO.md`, `git blame`, and
the individual feature commits — the **Done** section below is just a readable
summary (newest last).
## Conventions
- **Now** — items actively being worked on this session (only a few).
- **Next** — queued, well-scoped items.
- **Later** — idea backlog. Ideas marked **NOT NOW** were evaluated and rejected
for the stated reason; revisit only if circumstances change.
- When a task is completed: move it from Now/Next into **Done** (dated one-line)
in the same commit that finishes the work.
## Now
- (none — Tier 1 shipped: `pos system health`, `lib/notify.sh`, digest timer)
## Next
- Wire alerting into more tools as they are added (default: source
`lib/notify.sh`, call `notify_send` on success/failure).
## Later — idea backlog
- **Tier 2: watch plugins** — `pos system watch <event>`: poll conditions and
alert on change (public IP changed, disk > 90%, backup skipped, fail2ban
spike). Reuses `notify_send` + a systemd timer per watch.
- **Tier 2: `pos health` extras** — temperature/fan/load average thresholds,
`ss -tln` port checks for known services, SMART status for disks.
- **Tier 3: backup rotation + remote target** — keep-N rotations, upload to
rclone remote after verify, `--remote` flag, digest reports rotation age.
- **Tier 3: `pos secret` vault** — gpg/age-encrypted key-value store; backend
for future tools that need stored tokens.
- **Tier 3: `pos inventory`** — machine manifest (OS, packages, services,
mounted disks, USB devices) exportable as markdown/JSON.
- **Tier 4: `pos self update`** — pull repo, `make gen && make check`,
re-run install.sh to refresh `/usr/local/bin`.
- **Tier 4: `pos new`** — scaffold a new tool from `templates/pos-tool.sh`
(category, name, POS header, exec bit, doc stubs).
- **NOT NOW:** Telegram inbound bot (command handling) — outbound alerting
covers current needs; revisit if remote control is wanted.
- **NOT NOW:** per-category `bin/` subdirectories — flat `bin/` + filename
dispatch scales fine; revisit only if `bin/` passes ~40 files.
- **NOT NOW:** split `lib/entertainment-lib.sh` — fine under 600 lines; revisit
if it grows.
## Done (summary, newest last)
- 2026-08-06: Tier 1 — `pos system health` (dashboard + `--send`), `lib/notify.sh`
(wired into backup + firewall), daily digest timer via postinstall.
- 2026-08-06: Document Map index + Entertainment section in AGENT_Context (cf36780).
- 2026-08-06: Entertainment module — plugins (weather/joke/gold), `pos
entertainment config/enable/disable/send/status`, auto-trigger + Telegram send.
- 2026-08-05: `pos communication telegram` — `--parse-mode` (plain/markdown/html).
- 2026-08-05: doc/code sync gate — `make gen` + `make check` + pre-commit hook.
- 2026-08-05: `pos usb server` — USB Redirector control tool (494eae2).
- 2026-08-05: `pos <category> --help` auto-discovery in the dispatcher.
- 2026-08-05: AGENTS.md with lazy-loaded DOC references.
+19 -16
View File
@@ -10,19 +10,19 @@
<!-- GEN:START docmap --> <!-- GEN:START docmap -->
| ## 1. Project Overview | 2843 | | ## 1. Project Overview | 2843 |
| ## 2. Directory Structure | 44164 | | ## 2. Directory Structure | 44165 |
| ## 3. Installation Flow | 165216 | | ## 3. Installation Flow | 166217 |
| ## 4. The `pos` CLI System | 217273 | | ## 4. The `pos` CLI System | 218275 |
| ## 5. Shared Library — `lib/common.sh` | 274304 | | ## 5. Shared Library — `lib/common.sh` | 276306 |
| ## 6. Docker Compose / ScaleTail | 305347 | | ## 6. Docker Compose / ScaleTail | 307349 |
| ## 7. Optional Apps (`apps/`) | 348377 | | ## 7. Optional Apps (`apps/`) | 350379 |
| ## 8. Entertainment Module | 378391 | | ## 8. Entertainment Module | 380393 |
| ## 9. Systemd Services | 392402 | | ## 9. Systemd Services | 394404 |
| ## 10. Configuration Files | 403426 | | ## 10. Configuration Files | 405428 |
| ## 11. Coding Conventions | 427459 | | ## 11. Coding Conventions | 429461 |
| ## 12. Development Workflow | 460511 | | ## 12. Development Workflow | 462513 |
| ## 13. Key File Quick Reference | 512553 | | ## 13. Key File Quick Reference | 514556 |
| ## 14. Common Tasks for Agents | 554578 | | ## 14. Common Tasks for Agents | 557581 |
<!-- GEN:END docmap --> <!-- GEN:END docmap -->
## 1. Project Overview ## 1. Project Overview
@@ -76,6 +76,7 @@ Linux_post_install/
│ ├── pos-ssh-load-keys # Load all SSH keys into the agent │ ├── pos-ssh-load-keys # Load all SSH keys into the agent
│ ├── pos-system-backup # Encrypted (AES-256) folder snapshots (tar + gpg) │ ├── pos-system-backup # Encrypted (AES-256) folder snapshots (tar + gpg)
│ ├── pos-system-firewall # Interactive UFW management │ ├── pos-system-firewall # Interactive UFW management
│ ├── pos-system-health # Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL
│ ├── pos-usb-server # USB Redirector server control (--ls, --share; prompts when args omitted) │ ├── pos-usb-server # USB Redirector server control (--ls, --share; prompts when args omitted)
<!-- GEN:END tree --> <!-- GEN:END tree -->
│ ├── flag-reader # Inspect feature flags (list/status/--raw) │ ├── flag-reader # Inspect feature flags (list/status/--raw)
@@ -252,6 +253,7 @@ All non-interactive `pos` commands log output to `~/.local/share/linux_post_inst
| ssh | load-keys | `pos-ssh-load-keys` | Load all SSH keys into the agent | | ssh | load-keys | `pos-ssh-load-keys` | Load all SSH keys into the agent |
| system | backup | `pos-system-backup` | Encrypted (AES-256) folder snapshots (tar + gpg) | | system | backup | `pos-system-backup` | Encrypted (AES-256) folder snapshots (tar + gpg) |
| system | firewall | `pos-system-firewall` | Interactive UFW management | | system | firewall | `pos-system-firewall` | Interactive UFW management |
| system | health | `pos-system-health` | Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL |
| usb | server | `pos-usb-server` | USB Redirector server control (--ls, --share; prompts when args omitted) | | usb | server | `pos-usb-server` | USB Redirector server control (--ls, --share; prompts when args omitted) |
<!-- GEN:END dispatch --> <!-- GEN:END dispatch -->
@@ -542,10 +544,11 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
| `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location | | `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location |
| `bin/pos-network-scan` | 271 | Parallel ping sweep of CIDR | | `bin/pos-network-scan` | 271 | Parallel ping sweep of CIDR |
| `bin/pos-ssh-load-keys` | 31 | Load all SSH keys into the agent | | `bin/pos-ssh-load-keys` | 31 | Load all SSH keys into the agent |
| `bin/pos-system-backup` | 117 | Encrypted (AES-256) folder snapshots (tar + gpg) | | `bin/pos-system-backup` | 121 | Encrypted (AES-256) folder snapshots (tar + gpg) |
| `bin/pos-system-firewall` | 285 | Interactive UFW management | | `bin/pos-system-firewall` | 291 | Interactive UFW management |
| `bin/pos-system-health` | 230 | Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL |
| `bin/pos-usb-server` | 218 | USB Redirector server control (--ls, --share; prompts when args omitted) | | `bin/pos-usb-server` | 218 | USB Redirector server control (--ls, --share; prompts when args omitted) |
| `completions/pos.bash` | 188 | Dynamic bash completion | | `completions/pos.bash` | 189 | Dynamic bash completion |
<!-- GEN:END filetable --> <!-- GEN:END filetable -->
| `apps/install.sh` | 171 | App install/uninstall picker/orchestrator | | `apps/install.sh` | 171 | App install/uninstall picker/orchestrator |
+12
View File
@@ -275,6 +275,18 @@ Place it in `apps/<category>/<name>.sh`. It auto-appears in the picker — no re
## Best Practices ## Best Practices
### Alerting
To notify on events (Telegram), source the shared helper instead of calling the telegram tool directly:
```bash
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
notify_send "Backup completed"
notify_send "**disk full**" --markdown
```
`notify_send` is deliberately dependency-free (defines only itself, so it never clobbers a tool's own `log`/`warn`/`err`) and **silent-fails**: if Telegram is missing or not configured it warns and returns 0, never breaking the caller's flow or exit code. Source it opt-in in any tool that should alert; for failure alerts use `trap 'notify_send "..." ERR'`.
### Idempotency ### Idempotency
Check before creating, use `>>` with grep guards, don't overwrite user configs. Check before creating, use `>>` with grep guards, don't overwrite user configs.
+5 -2
View File
@@ -151,9 +151,12 @@ The standalone `vbox` command still works and forwards to `pos docker vbox` (see
| Command | File | Purpose | Configuration | | Command | File | Purpose | Configuration |
|---------|------|---------|---------------| |---------|------|---------|---------------|
| `sudo pos system firewall` | `bin/pos-system-firewall` | Interactive UFW ("UFW POWER") menu: add/delete rules, status, enable/disable/reset, default policies | Must run as root. Every command is previewed and confirmed before execution; supports `--dry-run`; keeps a history of executed commands | | `sudo pos system firewall` | `bin/pos-system-firewall` | Interactive UFW ("UFW POWER") menu: add/delete rules, status, enable/disable/reset, default policies | Must run as root. Every command is previewed and confirmed before execution; supports `--dry-run`; keeps a history of executed commands. Executed mutating changes are announced via `lib/notify.sh` |
| `pos system backup <folder-path>` | `bin/pos-system-backup` | Create a gpg-encrypted (AES-256) `tar.gz` snapshot of a folder and verify it | Prompts twice for a password (never stored). Uses `sudo tar`; needs `gnupg` (in `preinstall.sh` PACKAGES). Artifact `<name>_<date>.tar.gz.gpg` in the current directory, `chmod 600` | | `pos system backup <folder-path>` | `bin/pos-system-backup` | Create a gpg-encrypted (AES-256) `tar.gz` snapshot of a folder and verify it | Prompts twice for a password (never stored). Uses `sudo tar`; needs `gnupg` (in `preinstall.sh` PACKAGES). Artifact `<name>_<date>.tar.gz.gpg` in the current directory, `chmod 600`. Success/failure are announced via `lib/notify.sh` |
| `pos system backup --service` | `bin/pos-system-backup` | Lists folders under `/srv` and `~/srv`, lets you pick one, then runs the same backup | Roots via `BACKUP_SERVICE_ROOTS` (space-separated, default `/srv $HOME/srv`) | | `pos system backup --service` | `bin/pos-system-backup` | Lists folders under `/srv` and `~/srv`, lets you pick one, then runs the same backup | Roots via `BACKUP_SERVICE_ROOTS` (space-separated, default `/srv $HOME/srv`) |
| `pos system health [--send] [--markdown]` | `bin/pos-system-health` | Host health dashboard: disk per mount, RAM/swap, failed systemd units, backup age, fail2ban, docker containers. Exits 1 if any check FAILs | `--send`/`--markdown` send the summary via Telegram (`lib/notify.sh`). Backup age threshold via `HEALTH_BACKUP_MAX_AGE_DAYS` (default 2); backup search roots via `BACKUP_SERVICE_ROOTS` |
`systemd/pos-health.service` + `systemd/pos-health.timer` run `pos system health --send --markdown` daily at 08:00 as the installing user. `postinstall.sh` enables the timer automatically once `~/.config/linux_post_install/telegram.env` exists — re-run postinstall after configuring Telegram to pick it up.
### ssh ### ssh
+32 -3
View File
@@ -5,6 +5,7 @@ The units installed and enabled by `postinstall.sh`, plus the `pos` bash complet
- [Services](#services) - [Services](#services)
- [`autostart.service`](#autostartservice) - [`autostart.service`](#autostartservice)
- [`ssh-agent.service`](#ssh-agentservice) - [`ssh-agent.service`](#ssh-agentservice)
- [`pos-health.service`](#pos-healthservice)
- [Feature-flag gating](#feature-flag-gating) - [Feature-flag gating](#feature-flag-gating)
- [Bash completion](#bash-completion) - [Bash completion](#bash-completion)
@@ -12,7 +13,7 @@ The units installed and enabled by `postinstall.sh`, plus the `pos` bash complet
## Services ## Services
`postinstall.sh` copies every `systemd/*.service` to `/etc/systemd/system/`, runs `systemctl daemon-reload`, then enables each one (see the gating rule below). `postinstall.sh` copies every `systemd/*.service` (and `systemd/*.timer`) to `/etc/systemd/system/`, runs `systemctl daemon-reload`, then enables each one (see the gating rule below).
### autostart.service ### autostart.service
@@ -59,20 +60,48 @@ WantedBy=multi-user.target
**Configuration:** socket at `/run/ssh-agent/socket` (world-readable/writable). `~/.bashrc` (set by `postinstall.sh`) exports `SSH_AUTH_SOCK` to it. Not gated on any feature flag. **Configuration:** socket at `/run/ssh-agent/socket` (world-readable/writable). `~/.bashrc` (set by `postinstall.sh`) exports `SSH_AUTH_SOCK` to it. Not gated on any feature flag.
### pos-health.service
**Purpose:** daily "health digest" — runs `pos system health --send --markdown` at 08:00 and sends the report to Telegram.
```ini
[Unit]
Description=POS Health digest (daily report via Telegram)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=__POS_USER__
ExecStart=/usr/local/bin/pos system health --send --markdown
[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true
```
The service is `Type=oneshot` and is driven **only** by its companion `pos-health.timer` (`WantedBy=timers.target`); the service itself is never enabled directly.
**Configuration:** `postinstall.sh` substitutes `__POS_USER__` with the installing user (`${SUDO_USER:-$USER}`) so the digest uses that user's real Telegram config. The timer is enabled only when `~/.config/linux_post_install/telegram.env` already exists — otherwise postinstall warns and skips; re-run postinstall after configuring Telegram (`pos communication telegram config set TELEGRAM_*`) to install it.
--- ---
## Feature-flag gating ## Feature-flag gating
The systemd loop in `postinstall.sh` special-cases `autostart.service`: The systemd loop in `postinstall.sh` special-cases two units:
```bash ```bash
if [ "$svc_name" = "autostart.service" ] && ! flag_is_set autostart; then if [ "$svc_name" = "autostart.service" ] && ! flag_is_set autostart; then
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)" warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
continue continue
fi fi
if [ "$svc_name" = "pos-health.service" ]; then
# substitute User= and enable pos-health.timer only if Telegram is configured
fi
``` ```
Set the flag with `./install.sh --feature` (or `flag-set autostart`). See [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags). - `autostart.service` is **enabled** only when the `autostart` feature flag is set (`./install.sh --feature` or `flag-set autostart`). See [SCRIPTS.md → lib/flags.sh](SCRIPTS.md#libflagssh--feature-flags).
- `pos-health.service` is **not** enabled at all — `postinstall.sh` enables `pos-health.timer` instead, and only when a Telegram config already exists.
--- ---
+1 -1
View File
@@ -15,7 +15,7 @@ It is a **personal toolkit** — a bootstrap script, a unified `pos` CLI for eve
- Wi-Fi hotspot tools (`create_ap`, `wihotspot-gui`) via `pos network hotspot` - Wi-Fi hotspot tools (`create_ap`, `wihotspot-gui`) via `pos network hotspot`
- 15 optional desktop apps (VS Code, Brave, OBS, Tailscale, …) — pick what you want - 15 optional desktop apps (VS Code, Brave, OBS, Tailscale, …) — pick what you want
- 119+ self-hosted services with Tailscale access (Jellyfin, Home Assistant, …) - 119+ self-hosted services with Tailscale access (Jellyfin, Home Assistant, …)
- systemd services for SSH agent and boot-time automation - systemd services for SSH agent, boot-time automation, and a daily Telegram health digest (`pos system health`)
- Everything lands in `/usr/local/bin/` — you can delete the repo after install - Everything lands in `/usr/local/bin/` — you can delete the repo after install
## Quick Start ## Quick Start
+4
View File
@@ -4,6 +4,9 @@ set -euo pipefail
# POS_FLAGS: --service # POS_FLAGS: --service
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
trap 'notify_send "Backup FAILED: ${FOLDER:-unknown}"' ERR
usage() { usage() {
cat <<EOF cat <<EOF
@@ -115,3 +118,4 @@ gpg --batch --quiet --passphrase "$PASS" --decrypt "$ARCHIVE" | tar -tzf - > /de
unset PASS unset PASS
echo echo
log "Backup completed: $ARCHIVE" log "Backup completed: $ARCHIVE"
notify_send "Backup completed: $ARCHIVE"
+6
View File
@@ -9,6 +9,8 @@ if [[ $EUID -ne 0 ]]; then
exit 1 exit 1
fi fi
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
HISTORY=() HISTORY=()
DRY_RUN=0 DRY_RUN=0
@@ -29,6 +31,10 @@ run_cmd() {
echo "(dry-run) skipping execution" echo "(dry-run) skipping execution"
else else
"${cmd[@]}" "${cmd[@]}"
# Alert on mutating commands only (status queries are read-only)
if [[ "${cmd[1]:-}" != "status" ]]; then
notify_send "Firewall: ${cmd[*]}"
fi
fi fi
HISTORY+=("${cmd[*]}") HISTORY+=("${cmd[*]}")
else else
+230
View File
@@ -0,0 +1,230 @@
#!/usr/bin/env bash
set -euo pipefail
# POS: system health — Host health dashboard (disk, RAM, services, backup age, fail2ban, docker); exit 1 if any FAIL
# POS_FLAGS: --send --markdown
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
SEND=0
MARKDOWN=0
for arg in "$@"; do
case "$arg" in
-h|--help) usage_placeholder=1 ;;
--send) SEND=1 ;;
--markdown) MARKDOWN=1 ;;
*) err "Unknown option '$arg' (see --help)" ;;
esac
done
if [ "${usage_placeholder:-0}" -eq 1 ]; then
cat <<EOF
Usage: pos system health [--send] [--markdown]
Host health dashboard: disk, RAM/swap, failed systemd units, backup age,
fail2ban, docker containers. Exits 1 if any check FAILs.
Flags:
--send Send the summary via Telegram (uses lib/notify.sh)
--markdown Same as --send, with markdown parse mode
-h, --help Show this help
Environment:
HEALTH_BACKUP_MAX_AGE_DAYS Max backup age before a WARN (default: 2)
BACKUP_SERVICE_ROOTS Where to look for backups (default: /srv \$HOME/srv)
Examples:
pos system health
pos system health --send
EOF
exit 0
fi
if [ "$MARKDOWN" -eq 1 ]; then
SEND=1
fi
FAILURES=0
WARNINGS=0
REPORT=()
report() {
local st="$1" label="$2" detail="$3" line
case "$st" in
ok)
echo " ${GREEN}${BOLD}[ OK ]${RESET} ${label}: ${detail}"
printf -v line "[ OK ] %s: %s" "$label" "$detail"
;;
warn)
echo " ${YELLOW}${BOLD}[WARN]${RESET} ${label}: ${detail}"
printf -v line "[WARN] %s: %s" "$label" "$detail"
WARNINGS=$((WARNINGS + 1))
;;
fail)
echo " ${RED}${BOLD}[FAIL]${RESET} ${label}: ${detail}"
printf -v line "[FAIL] %s: %s" "$label" "$detail"
FAILURES=$((FAILURES + 1))
;;
esac
REPORT+=("$line")
}
host="$(hostname -s 2>/dev/null || echo "localhost")"
ip_public="$(curl -fsS -m 5 https://api.ipify.org 2>/dev/null || echo "unreachable")"
loadavg="$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo "?")"
uptime_s="$(uptime -p 2>/dev/null || echo "?")"
section "System Health — ${host}"
echo " ${BLUE}${BOLD}date${RESET} $(date '+%Y-%m-%d %H:%M:%S')"
echo " ${BLUE}${BOLD}uptime${RESET} ${uptime_s}"
echo " ${BLUE}${BOLD}load${RESET} ${loadavg}"
echo " ${BLUE}${BOLD}ip${RESET} ${ip_public}"
echo
# ── Disk ──────────────────────────────────────────────────────
DISK_PCT_MAX=0
DISK_ISSUES=()
while read -r fs use mount; do
pct=${use%\%}
if [ "$pct" -gt "$DISK_PCT_MAX" ]; then
DISK_PCT_MAX="$pct"
fi
if [ "$pct" -ge 90 ]; then
DISK_ISSUES+=("${mount} ${pct}%")
elif [ "$pct" -ge 85 ]; then
DISK_ISSUES+=("${mount} ${pct}%")
fi
done < <(df -P -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs -x proc -x sysfs -x cgroup2 -x zfs -x aufs 2>/dev/null | awk 'NR>1 && $1 !~ /loop|sr[0-9]/ {print $1, $5, $6}')
if [ ${#DISK_ISSUES[@]} -gt 0 ]; then
report fail "disk" "${DISK_ISSUES[*]}"
else
report ok "disk" "max usage ${DISK_PCT_MAX}%"
fi
# ── RAM / swap ────────────────────────────────────────────────
mem=($(free -m | awk '/^Mem:/{print $2, $7}'))
swap=($(free -m | awk '/^Swap:/{print $2, $3}'))
MEM_TOTAL=${mem[0]:-0}
MEM_AVAIL=${mem[1]:-0}
MEM_PCT=100
if [ "$MEM_TOTAL" -gt 0 ]; then
MEM_PCT=$(( (MEM_TOTAL - MEM_AVAIL) * 100 / MEM_TOTAL ))
fi
SWAP_PCT=0
if [ "${swap[0]:-0}" -gt 0 ]; then
SWAP_PCT=$(( swap[1] * 100 / swap[0] ))
fi
if [ "$MEM_PCT" -ge 95 ] || [ "$SWAP_PCT" -ge 90 ]; then
report fail "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
elif [ "$MEM_PCT" -ge 85 ]; then
report warn "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
else
report ok "ram" "mem ${MEM_PCT}% used, swap ${SWAP_PCT}%"
fi
# ── Failed systemd units ──────────────────────────────────────
if command -v systemctl >/dev/null 2>&1; then
failed_units=()
while IFS= read -r u; do
[ -n "$u" ] && failed_units+=("${u%% *}")
done < <(systemctl --failed --plain --no-legend --no-pager 2>/dev/null || true)
if [ "${#failed_units[@]}" -gt 0 ]; then
report fail "services" "${#failed_units[@]} failed unit(s): ${failed_units[*]}"
else
report ok "services" "no failed units"
fi
else
report warn "services" "systemd not available"
fi
# ── Backup age ────────────────────────────────────────────────
MAX_AGE="${HEALTH_BACKUP_MAX_AGE_DAYS:-2}"
roots=()
if [ -n "${BACKUP_SERVICE_ROOTS:-}" ]; then
read -r -a roots <<< "$BACKUP_SERVICE_ROOTS"
else
roots=(/srv "$HOME/srv")
fi
newest=""
newest_ts=0
for root in "${roots[@]}"; do
[ -d "$root" ] || continue
while IFS= read -r line; do
ts="${line%% *}"
if [ "${ts%%.*}" -gt "$newest_ts" ]; then
newest_ts="${ts%%.*}"
newest="${line#* }"
fi
done < <(find "$root" -maxdepth 2 -type f -name '*.tar.gz.gpg' -printf '%T@ %p\n' 2>/dev/null || true)
done
if [ -z "$newest" ]; then
report warn "backup" "no backups found under ${roots[*]}"
else
now="$(date +%s)"
days=$(( (now - newest_ts) / 86400 ))
if [ "$days" -gt "$MAX_AGE" ]; then
report warn "backup" "${days}d old: ${newest}"
else
report ok "backup" "${days}d old: ${newest}"
fi
fi
# ── fail2ban ──────────────────────────────────────────────────
if command -v fail2ban-client >/dev/null 2>&1; then
if fail2ban-client ping >/dev/null 2>&1; then
banned=0
jails=$(fail2ban-client status 2>/dev/null | awk -F': *' '/Jail list/{print $2}')
for j in $jails; do
n=$(fail2ban-client status "$j" 2>/dev/null | awk -F': *' '/Total banned/{print $2}')
n="${n:-0}"
banned=$(( banned + (n > 0 ? n : 0) ))
done
report ok "fail2ban" "running, ${banned} total banned across ${jails:-0} jail(s)"
else
report warn "fail2ban" "installed but not running"
fi
fi
# ── Docker ────────────────────────────────────────────────────
if command -v docker >/dev/null 2>&1; then
bad_containers=()
running=0
while IFS= read -r name state; do
case "$state" in
restarting|unhealthy) bad_containers+=("${name}(${state})") ;;
esac
done < <(docker ps -a --format '{{.Names}} {{.State}}' 2>/dev/null || true)
if [ "${#bad_containers[@]}" -gt 0 ]; then
report fail "docker" "${#bad_containers[@]} problem container(s): ${bad_containers[*]}"
else
report ok "docker" "no unhealthy/restarting containers"
fi
fi
echo
if [ "$FAILURES" -gt 0 ]; then
verdict="${FAILURES} FAIL, ${WARNINGS} WARN"
echo " ${RED}${BOLD}VERDICT${RESET} ${verdict}"
elif [ "$WARNINGS" -gt 0 ]; then
verdict="${WARNINGS} WARN — mostly healthy"
echo " ${YELLOW}${BOLD}VERDICT${RESET} ${verdict}"
else
verdict="all good"
echo " ${GREEN}${BOLD}VERDICT${RESET} ${verdict}"
fi
if [ "$SEND" -eq 1 ]; then
msg="POS Health — ${host}
Verdict: ${verdict}"
for line in "${REPORT[@]}"; do
msg+="
${line}"
done
if [ "$MARKDOWN" -eq 1 ]; then
notify_send "$msg" --markdown
else
notify_send "$msg"
fi
fi
[ "$FAILURES" -eq 0 ]
+1
View File
@@ -7,6 +7,7 @@ _pos_flags[communication-telegram]="--send --type --caption --parse-mode --no-pr
_pos_flags[entertainment-send]="--print --markdown" _pos_flags[entertainment-send]="--print --markdown"
_pos_flags[network-hotspot]="--foreground" _pos_flags[network-hotspot]="--foreground"
_pos_flags[system-backup]="--service" _pos_flags[system-backup]="--service"
_pos_flags[system-health]="--send --markdown"
_pos_flags[usb-server]="--ls --ls-shared --share --unshare --auto-share --callback --close-callback --auto-connect --disconnect --nickname --timeout --port --info --version" _pos_flags[usb-server]="--ls --ls-shared --share --unshare --auto-share --callback --close-callback --auto-connect --disconnect --nickname --timeout --port --info --version"
# GEN:END posflags # GEN:END posflags
+1
View File
@@ -113,6 +113,7 @@ if should_run 2 scripts; then
done done
run sudo install -m 644 lib/common.sh /usr/local/bin/common.sh run sudo install -m 644 lib/common.sh /usr/local/bin/common.sh
run sudo install -m 644 lib/flags.sh /usr/local/bin/flags.sh run sudo install -m 644 lib/flags.sh /usr/local/bin/flags.sh
run sudo install -m 644 lib/notify.sh /usr/local/bin/notify.sh
run sudo install -m 644 lib/entertainment-lib.sh /usr/local/bin/entertainment-lib.sh run sudo install -m 644 lib/entertainment-lib.sh /usr/local/bin/entertainment-lib.sh
# ── Entertainment plugins ──────────────────────────────── # ── Entertainment plugins ────────────────────────────────
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# lib/notify.sh — optional alerting helper. Self-contained by design:
# defines ONLY notify_send() so it can be sourced by tools that define
# their own log/warn/err (e.g. pos-system-firewall) without clobbering.
#
# Usage (opt-in — source it, do NOT auto-load from common.sh):
# source "$(dirname "$0")/../lib/notify.sh" 2>/dev/null || source "$(dirname "$0")/notify.sh"
# notify_send "Backup completed: $ARCHIVE"
# notify_send "⚠ disk full" --markdown
#
# Delegates to `pos communication telegram send`; silent-fail if the
# telegram sender is missing or not configured (warns, never breaks the
# caller and never changes its exit code).
notify_send() {
local msg="" markdown=0
while [ $# -gt 0 ]; do
case "$1" in
--markdown) markdown=1; shift ;;
*) msg="$1"; shift ;;
esac
done
if [ -z "$msg" ]; then
warn "notify_send: empty message, skipped" 2>/dev/null || echo "notify_send: empty message, skipped"
return 0
fi
local tg
tg="$(command -v pos-communication-telegram 2>/dev/null)" || \
tg="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../bin/pos-communication-telegram"
if [ ! -x "$tg" ]; then
warn "notify_send: pos-communication-telegram not found, notification skipped" 2>/dev/null || true
return 0
fi
if [ "$markdown" -eq 1 ]; then
"$tg" send "$msg" --parse-mode markdown >/dev/null 2>&1 || {
warn "notify_send: telegram send failed, notification skipped" 2>/dev/null || true
}
else
"$tg" send "$msg" >/dev/null 2>&1 || {
warn "notify_send: telegram send failed, notification skipped" 2>/dev/null || true
}
fi
}
+22
View File
@@ -94,8 +94,18 @@ fi
# ── systemd services ─────────────────────────────────────────── # ── systemd services ───────────────────────────────────────────
if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then
run sudo cp systemd/*.service /etc/systemd/system/ run sudo cp systemd/*.service /etc/systemd/system/
if [ -n "$(ls -A systemd/*.timer 2>/dev/null)" ]; then
run sudo cp systemd/*.timer /etc/systemd/system/
fi
run sudo systemctl daemon-reload run sudo systemctl daemon-reload
# pos-health.service runs as the installing user so the digest uses that
# user's real Telegram config. The timer is enabled only when a telegram.env
# is configured — re-run postinstall after configuring Telegram to pick it up.
DIGEST_USER="${SUDO_USER:-$USER}"
DIGEST_HOME="$(getent passwd "$DIGEST_USER" 2>/dev/null | cut -d: -f6)"
DIGEST_HOME="${DIGEST_HOME:-$HOME}"
for svc in systemd/*.service; do for svc in systemd/*.service; do
svc_name=$(basename "$svc") svc_name=$(basename "$svc")
# autostart.service runs features/autostart.sh — enable only when # autostart.service runs features/autostart.sh — enable only when
@@ -104,6 +114,18 @@ if [ -d systemd ] && [ -n "$(ls -A systemd/*.service 2>/dev/null)" ]; then
warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)" warn "autostart feature not installed — skipping autostart.service (run ./install.sh --feature)"
continue continue
fi fi
if [ "$svc_name" = "pos-health.service" ]; then
run sudo sed -i "s|__POS_USER__|$DIGEST_USER|" "/etc/systemd/system/$svc_name"
run sudo systemctl daemon-reload
if [ -f "$DIGEST_HOME/.config/linux_post_install/telegram.env" ]; then
run sudo systemctl enable --now pos-health.timer 2>/dev/null || \
run sudo systemctl enable pos-health.timer
log "Daily health digest timer enabled for $DIGEST_USER"
else
warn "Telegram not configured — skipping health digest timer (run 'pos communication telegram config set ...' then re-run postinstall)"
fi
continue
fi
run sudo systemctl enable --now "$svc_name" 2>/dev/null || \ run sudo systemctl enable --now "$svc_name" 2>/dev/null || \
run sudo systemctl enable "$svc_name" run sudo systemctl enable "$svc_name"
done done
+9
View File
@@ -0,0 +1,9 @@
[Unit]
Description=POS Health digest (daily report via Telegram)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=__POS_USER__
ExecStart=/usr/local/bin/pos system health --send --markdown
+9
View File
@@ -0,0 +1,9 @@
[Unit]
Description=Daily POS health digest timer
[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true
[Install]
WantedBy=timers.target