diff --git a/DOC/POS.md b/DOC/POS.md index eb4eb37..b22935c 100644 --- a/DOC/POS.md +++ b/DOC/POS.md @@ -131,10 +131,11 @@ Runs a persistent `aria2c` JSON-RPC daemon (`localhost:6800`) as a **systemd use | `pos network download watch [gid]` | Live table, 2 s refresh; with a GID it exits when that download completes | | `pos network download restart ` | Re-queue a finished/errored download from history: torrents re-add via magnet (info-hash + trackers), HTTP via their original URLs — `--continue=true` resumes partial files, a complete file re-verifies instantly. Options `--dir`, `--seed`, `--split`, `--tmux` | | `pos network download retry ` | Smart retry of errored downloads: waits out internet outages (poll `--interval`, give up after `--max-wait`), then re-queues and re-verifies. Sources failing with aria2 error 3 are marked permanent in `~/.config/linux_post_install/download.retry` (id = `url:` / `bt:`) and skipped by `retry all` — manual `restart` overrides. `--once` (healer timer mode) skips the wait and exits 0 even on failure; `--quiet` silences output. Options `--dir`, `--seed`, `--split`, `--tmux` | +| `pos network download replace ` | Give a dead download a fresh URL: re-queues with the same `dir` + file name (so `--continue=true` resumes the partial), forgets the old dead source from `download.retry`, and verifies the new link — a dead replacement is diagnosed and marked permanent. `status` lists downloads needing this. Single-file HTTP/FTP only (torrents: `restart`); options `--dir`, `--split`, `--tmux` | **`--tmux`:** after enqueueing, `add`/`torrent`/`metalink` open a detached tmux session `dl-` running `watch ` (name from `--out` or the URL basename, sanitized and truncated to 40 chars; `-2` suffix on collision). The session closes itself when the download finishes — attach with `tmux attach -t dl-`. -**Outage resilience:** `watch ` auto-restarts its download when the network comes back (it polls `NET_PROBE`, default `timeout 3 bash -c '` auto-restarts its download when the network comes back (it polls `NET_PROBE`, default `timeout 3 bash -c ' ()` line and `replace ` resumes it with a new URL. Both are dry-run aware. ### docker @@ -214,7 +215,7 @@ The standalone `vbox` command still works and forwards to `pos docker vbox` (see | `pos system backup ` | `bin/pos-system-backup` | Create a gpg-encrypted (AES-256) `tar.gz` snapshot of a folder and verify it | Prompts twice for a password (never stored). Uses `sudo tar`; needs `gnupg` (in `preinstall.sh` PACKAGES). Artifact `_.tar.gz.gpg` in the current directory, `chmod 600`. Success/failure are announced via `lib/notify.sh` | | `pos system backup --service` | `bin/pos-system-backup` | Lists folders under `/srv` and `~/srv`, lets you pick one, then runs the same backup | Roots via `BACKUP_SERVICE_ROOTS` (space-separated, default `/srv $HOME/srv`) or `~/.config/linux_post_install/system.env` | | `pos system health [--send] [--markdown]` | `bin/pos-system-health` | Host health dashboard: disk per mount, RAM/swap, failed systemd units, backup age, fail2ban, docker containers. Exits 1 if any check FAILs | `--send`/`--markdown` are notification-only: they send the summary via `lib/notify.sh` to every platform in `NOTIFY_PLATFORM` and do NOT print the dashboard (so wrappers like the Telegram listener don't echo it back — pair with the listener's `@quiet` marker). `HEALTH_BACKUP_MAX_AGE_DAYS` (default 2) and `BACKUP_SERVICE_ROOTS` come from `~/.config/linux_post_install/system.env`; `--help` shows the effective values. Platform list from `~/.config/linux_post_install/notify.env` | -| `pos system event-trigger ` | `bin/pos-system-event-trigger` | State-based rule monitors: `run`, `config`, `list`, `enable [interval]`, `disable`, `status`. Each line of `~/.config/linux_post_install/event.env` is an independent rule `["msg" if ] ` (op `> < >= <= == !=`, unit suffix ok: `60c`, `80%`); the check command's first numeric output is compared float-safe. Alerts once on false→true, plus one recovery message on true→false — no repeats while the condition holds, via `lib/notify.sh` | `config` is an interactive editor that validates rules by test-running the check; `run` is what the systemd user timer (`pos-event-trigger.timer` + oneshot `.service`, interval set at `enable`) executes; supports `--dry-run`; rules are arbitrary shell commands (chmod 600, same trust model as the Telegram map); template `config/event.env` auto-installed no-clobber by postinstall | +| `pos system schedule ` | `bin/pos-system-schedule` | Scheduled jobs — run a command on a timer, notify (or stay silent): `run [name\|all]`, `list`, `config`, `enable [name\|all]`, `disable [name\|all]`, `status`, `migrate`. Each job is a file in `~/.config/linux_post_install/schedule.d/.env` with `INTERVAL` (`5m…59m`, `1h…23h`, `hourly`, `daily`, `weekly`, `OnCalendar=…`), `NOTIFY` policy, optional `MSG`, `RULE` (threshold only), and `COMMAND` = the literal rest of the line (pipes/quotes/`sudo` fine). Policies: `always` (full output every run), `onchange` (send when output differs from the last run; first run always sends), `onerror` (non-zero exit or empty output), `threshold` (first numeric output vs `RULE`, alert on false→true + one recovery — the old event-trigger behavior), `never` (side-effect jobs, no notify) | One systemd **user** timer pair per job (`pos-schedule-.timer` + oneshot `.service`, `Persistent=true`), reconciled on `enable`/`disable`; the legacy single `pos-event-trigger` timer is auto-removed. `migrate` converts a pre-existing `event.env` rule set into `schedule.d/rule-N.env` threshold jobs. `config` is an interactive editor (add/edit/remove/enable/disable, validates interval + threshold); alerts via `lib/notify.sh`; `--dry-run` previews runs/writes/sends; jobs are arbitrary shell commands (chmod 600, same trust model as the Telegram map); starter jobs in `config/schedule.d/` auto-installed no-clobber by postinstall | `systemd/pos-health.service` + `systemd/pos-health.timer` run `pos system health --send --markdown` daily at 08:00 as the installing user. `postinstall.sh` enables the timer automatically once `~/.config/linux_post_install/telegram.env` exists — re-run postinstall after configuring a notify platform to pick it up. The service also loads `system.env` + `notify.env` via `EnvironmentFile=`. diff --git a/DOC/howto/network.md b/DOC/howto/network.md index daf9fd9..456bffa 100644 --- a/DOC/howto/network.md +++ b/DOC/howto/network.md @@ -173,6 +173,23 @@ on its own once a download started, so no shell is needed on headless boxes. remembered in `~/.config/linux_post_install/download.retry` and skipped by `retry all` — `pos network download restart ` re-queues them by hand. +**Recipe: dead link.** When a source is genuinely gone (404/410), `pos network +download status` flags it: + +``` + needs fresh link: ubuntu.iso (2e9dffc40e6a1b2c) — pos network download replace 2e9dffc40e6a1b2c +``` + +Give it a working URL — same dir and file name are kept, so the partial file +resumes instead of restarting from zero: + +```bash +pos network download replace 2e9dffc40e6a1b2c https://mirror.example.com/ubuntu.iso +``` + +The old dead source is forgotten, the new link is verified (a dead replacement +is reported and marked permanent instead of silently looping). + --- ## Related diff --git a/bin/pos-network-download b/bin/pos-network-download index 2dfe30a..bf08577 100755 --- a/bin/pos-network-download +++ b/bin/pos-network-download @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -euo pipefail # POS: network download — aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) -# POS_SUBCMDS: start stop status add torrent metalink list info files peers pause resume remove purge move limit set watch restart retry +# POS_SUBCMDS: start stop status add torrent metalink list info files peers pause resume remove purge move limit set watch restart retry replace # POS_FLAGS: --dir --out --split --seed --force --upload --gid --tmux source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" @@ -69,6 +69,8 @@ Commands: retry Smart retry: waits out internet outages, then re-queues and re-checks; reports real problems (dead source) instead of looping (--once: single check without waiting — used by the healer timer) + replace Give a dead download a fresh URL: same dir + file name, resumes + the partial file (status flags downloads that need this) Options (add / torrent / metalink): --dir download directory (default: $DOWNLOAD_DIR) @@ -93,9 +95,12 @@ Examples: pos network download watch pos network download restart 2e9dffc4 pos network download retry all + pos network download replace 2e9dffc4 https://mirror.example.com/ubuntu.iso Failed downloads are auto-retried by the 'retry healer' systemd timer while the daemon runs; it arms on download start and disables itself when nothing is left. +Sources that fail with a real 404/410 are marked permanent — 'status' flags them +with a fresh-link hint and 'replace ' resumes them with a new URL. Config: RPC secret in $CONFIG_FILE (chmod 600). Env: RPC_PORT, RPC_SECRET, DOWNLOAD_DIR, NET_PROBE override defaults (test seams). @@ -212,9 +217,26 @@ cmd_status() { if daemon_active; then rpc aria2.getGlobalStat | jq -r '"active: \(.result.numActive) waiting: \(.result.numWaiting) stopped: \(.result.numStopped) (history: \(.result.numStoppedTotal))"' echo "aria2: $(rpc aria2.getVersion | jq -r '.result.version')" + dead_source_advisory fi } +dead_source_advisory() { + # Stopped errored downloads whose source is marked permanently failing — + # they need a fresh link via 'replace '. + local ids + ids=$(grep . "$RETRY_STATE" 2>/dev/null || true) + [ -n "$ids" ] || return 0 + rpc aria2.tellStopped 0 200 | jq -r --arg ids "$ids" ' + .result[] | select(.status == "error") | + ((if .bittorrent then "bt:" + .bittorrent.infoHash else "url:" + (.files[0].uris[0].uri // .files[0].path // "?") end)) as $id | + select(($ids | split("\n")) | index($id)) | + [.gid, (if .bittorrent then (.bittorrent.info.name // "?") else (.files[0].path // "?" | split("/") | last) end)] | @tsv' \ + | while IFS=$'\t' read -r gid name; do + printf ' needs fresh link: %s (%s) — pos network download replace %s \n' "$name" "$gid" "$gid" + done +} + ensure_daemon() { if ! daemon_active; then log "daemon not running — starting it" @@ -434,6 +456,53 @@ cmd_restart() { if [ "$tmux" -eq 1 ]; then tmux_watch "${newgids[0]}" "$name"; fi } +cmd_replace() { # replace — fresh link for a dead download, same dir/file (resumes partial) + local gid="" url="" dir="" split="" tmux=0 data st nfiles old_uri odir name newgid oa d + while [ $# -gt 0 ]; do + case "$1" in + --dir) dir="${2:-}"; shift 2 ;; + --dir=*) dir="${1#*=}"; shift ;; + --split) split="${2:-}"; shift 2 ;; + --split=*) split="${1#*=}"; shift ;; + --tmux) tmux=1; shift ;; + -h|--help) usage ;; + -*) err "replace: unknown option: $1" ;; + *) if [ -z "$gid" ]; then gid="$1"; else url="$1"; fi; shift ;; + esac + done + [ -n "$gid" ] && [ -n "$url" ] || err "replace: usage: pos network download replace " + [ "$tmux" -eq 1 ] && { command -v tmux &>/dev/null || err "tmux not found (install tmux) — needed for --tmux"; } + dir="${dir/#\~/$HOME}" + ensure_daemon + gid=$(resolve_gid "$gid") + data=$(rpc aria2.tellStatus "$(json_str "$gid")") + st=$(printf '%s' "$data" | jq -r '.result.status') + case "$st" in + complete|error|removed) ;; + active) err "gid $gid is active — it's already downloading" ;; + *) err "gid $gid has status '$st' — cannot replace" ;; + esac + if [ "$(printf '%s' "$data" | jq -r 'if .result.bittorrent then 1 else 0 end')" = "1" ]; then + err "gid $gid is a torrent — a fresh URL can't replace it (use: restart $gid to re-add the same magnet)" + fi + nfiles=$(printf '%s' "$data" | jq -r '.result.files | length') + [ "$nfiles" -eq 1 ] || err "gid $gid has $nfiles files — replace supports single-file downloads only" + old_uri=$(printf '%s' "$data" | jq -r '.result.files[0].uris[0].uri // ""') + name=$(download_name "$data"); [ -n "$name" ] || name="download" + odir=$(printf '%s' "$data" | jq -r '.result.dir // ""') + d="${dir:-$odir}" + oa=() + [ -n "$d" ] && oa+=("dir=$d") + oa+=("out=$name") + [ -n "$split" ] && oa+=("split=$split") + newgid=$(rpc aria2.addUri "$(json_arr "$url")" "$(opts_json "${oa[@]}")" | jq -r '.result') + log "replaced $gid → $newgid: $name — re-queued with fresh link (resumes partial file)" + [ -n "$old_uri" ] && retry_unmark "url:$old_uri" + retry_verify "$newgid" "url:$url" || err "replacement link also failed — see messages above" + ensure_healer + if [ "$tmux" -eq 1 ]; then tmux_watch "$newgid" "$name"; fi +} + retry_src_id() { # stable identity for the permanent-failure list (survives new gids) local data data=$(rpc aria2.tellStatus "$(json_str "$1")") @@ -447,6 +516,12 @@ retry_mark_permanent() { retry_is_permanent "$1" || printf '%s\n' "$1" >> "$RETRY_STATE" } +retry_unmark() { # retry_unmark — drop one source from the permanent list + [ -f "$RETRY_STATE" ] || return 0 + grep -vxF "$1" "$RETRY_STATE" > "$RETRY_STATE.tmp" || true + mv "$RETRY_STATE.tmp" "$RETRY_STATE" +} + retry_verify() { # retry_verify → 0 ok / 1 failed (marks real failures permanent) local g="$1" src="$2" attempt=0 st ec em data while [ "$attempt" -lt 5 ]; do @@ -455,21 +530,21 @@ retry_verify() { # retry_verify → 0 ok / 1 failed (marks re data=$(rpc aria2.tellStatus "$(json_str "$g")" 2>/dev/null || true) st=$(printf '%s' "$data" | jq -r '.result.status // "?"') case "$st" in - active|waiting) [ "$quiet" -eq 1 ] || log "retry running — $g (${g:0:8})"; return 0 ;; - complete) [ "$quiet" -eq 1 ] || log "retry finished instantly — $g already on disk"; return 0 ;; + active|waiting) [ "${quiet:-0}" -eq 1 ] || log "retry running — $g (${g:0:8})"; return 0 ;; + complete) [ "${quiet:-0}" -eq 1 ] || log "retry finished instantly — $g already on disk"; return 0 ;; error|removed) ec=$(printf '%s' "$data" | jq -r '.result.errorCode // "?"') em=$(printf '%s' "$data" | jq -r '.result.errorMessage // "?"') if [ "$ec" = "3" ]; then retry_mark_permanent "$src" - [ "$quiet" -eq 1 ] || err "real problem — source is gone (aria2 error 3: $em). Marked permanent; manual 'restart' overrides." + [ "${quiet:-0}" -eq 1 ] || err "real problem — source is gone (aria2 error 3: $em). Marked permanent; manual 'restart' overrides." return 1 fi - [ "$quiet" -eq 1 ] || warn "retry attempt $attempt failed (aria2 error $ec: $em) — will re-check" + [ "${quiet:-0}" -eq 1 ] || warn "retry attempt $attempt failed (aria2 error $ec: $em) — will re-check" ;; esac done - [ "$quiet" -eq 1 ] || warn "retry of $g did not stabilize after 5 checks" + [ "${quiet:-0}" -eq 1 ] || warn "retry of $g did not stabilize after 5 checks" return 1 } @@ -864,7 +939,7 @@ main() { torrent) cmd_torrent "$@" ;; metalink) cmd_metalink "$@" ;; list) cmd_list ;; - info|files|peers|pause|resume|remove|move|limit|set|watch|restart|retry) + info|files|peers|pause|resume|remove|move|limit|set|watch|restart|retry|replace) "cmd_$cmd" "$@" ;; purge) cmd_purge ;; *) err "unknown command: $cmd (see 'pos network download --help')" ;;