feat: pos media ytsync — incremental YouTube channel/playlist sync into ~/Videos
This commit is contained in:
@@ -128,3 +128,4 @@ summary (newest last).
|
||||
- **2026-08-14** — `pos system backup` optional encryption (`--no-encrypt` flag + `BACKUP_ENCRYPT=0` env, flag-or-env — user chose "Flag + env only"): plain path keeps a verified `.tar.gz` with no password prompt (headless/cron safe); encrypt path unchanged (prompt → gpg AES-256 → decrypt-verify; the gpg dep-guard moved into the encrypt branch so plain backups no longer require `gnupg`). Arg parsing rewritten as a loop over `"$@"` so `pos system backup <folder> --no-encrypt` works with the flag after the folder; usage() documents all three forms + the plain artifact name; `# POS_FLAGS: --service --no-encrypt`; `config/system.env` template gains `#BACKUP_ENCRYPT=0`; POS.md row + howto/system.md section updated. Verified: stub suite +2 cases (T18 flag / T19 env: plain .tar.gz artifact, gpg never called via `$GPG_CALLED`, USB copy + sha256 of the plain archive, notify wording) — 65/65 green; `bash -n`, `make gen && make check`, `make lint` 0 FAIL / 0 WARN.
|
||||
- **2026-08-14** — CI green-check via plain git (no SSH to the runner, no API tokens — user chose "CI tags + git ls-remote" + "scripts/ci-status.sh helper"): `.gitea/workflows/lint.yml` scoped to `on: push: branches: [main]` (tag pushes no longer re-trigger it) and the gate step now reports its own outcome as a lightweight tag — `ci-ok/$GITHUB_SHA` on success / `ci-fail/$GITHUB_SHA` on failure, pushed over HTTP with the jobs automatic `GITEA_TOKEN` to `http://oauth2:${GITEA_TOKEN}@gitea.skink-platy.ts.net:3000/admin/Linux_post_install.git` (runner container already host-maps that hostname to 100.111.241.54); `steps.gates.conclusion` decides ok/fail, `if: always()` (guarded to `push` events) covers failed gate runs, and an existing-tag guard makes re-runs idempotent. New executable `scripts/ci-status.sh [--wait] [<sha>]` reads the tags via `git ls-remote` (origin, `CI_STATUS_REMOTE` override): GREEN (0) / RED (1) / PENDING (2); `--wait` polls every 10s up to 10 min. DEV.md §CI gains a "Checking green without SSH" bullet. Verified: `bash -n`, yaml-parse OK, `make gen && make check`, `make lint` 0 FAIL / 0 WARN; first live-tag verification pending the push (fallback if Gitea clamps token-push: PAT as workflow secret).
|
||||
- **2026-08-14** — `pos media sync` (`bin/pos-media-sync`) — incremental Music → USB sync, plus the shared USB layer it builds on. **New lib `lib/usb-lib.sh`** (194 lines, installed by install.sh): `usb_detect` (lsblk JSON, TRAN + lsusb/by-id cross-check → `USB_MOUNTED`/`USB_UNMOUNTED`), `usb_related_present`, `usb_mount_offer` (`/media/<label>` mount-offer, `usb-automount` scheme), `usb_pick_root <prefix> <subfolder> <giveup-msg>` (detect → mount-offer → single/multi picker → `USB_ROOT`); seams `USB_MOUNT_BASE`/`USB_BYID` with `BACKUP_MOUNT_BASE`/`BACKUP_USB_BYID` aliases so existing `system.env` lines keep working; TRAN-fallback warning deduped to once per scan. `pos-system-backup` refactored onto it (216 lines, was 364) — re-ran the backup stub suite: 65/65 green. **Sync tool**: add/update only, never deletes (user choice); `--mp3`/`--mp4` filter (neither = both), `--source <dir>` (default `MEDIA_SYNC_SOURCE`/`$HOME/Music`), `--dry-run` preview with counts; copies missing/changed (size/mtime) files into `<usb>/Music/` (`MEDIA_SYNC_DEST`) preserving the tree via `cp --preserve=timestamps`; result notified via `lib/notify.sh`; `media-sync` added to `INTERACTIVE_CMDS`; deps guards (`lsblk`/`jq`) before `-h|--help`. Docs: POS.md media row, howto/media.md section, SCRIPTS.md lib section + Phase-2 lib list, system.env seams, DEV.md env-seam registry, AGENT_Context Common Tasks + hand-maintained lib row (+usb-lib 194) + gen'd tree/dispatch/filetable/flags. Verified: new stub suite `/tmp/opencode/msync-run.sh` 46/46 green (fresh/no-op/update/filter/dry-run/multi-stick/mount-offer/no-USB skip/never-delete/--source/TRAN-fallback/notify) — caught and fixed an inverted `needs_copy` return; `make gen && make check`, `make lint` 0 FAIL / 0 WARN; dispatch via `pos media sync --help` + `pos media` listing.
|
||||
- **2026-08-22** — `pos media ytsync` (`bin/pos-media-ytsync`) — incremental YouTube channel/playlist sync into `~/Videos`, implemented per the Architect decisions D1–D9 + Designer UX contract (`reportAgents/2026-08-22-*.md`). Subcommands `add [url] / sync [name] / list / remove <name>` + `--dry-run`; bare invocation = interactive menu (`/dev/tty` reads, EOF-safe, NOT in INTERACTIVE_CMDS so dispatcher tee logging is kept; empty state goes straight to the URL prompt). One yt-dlp call per new video (`bestvideo*+bestaudio/best` → MP4, metadata/chapters/thumbnail, `--no-overwrites`, `--windows-filenames --trim-filenames 120`, retries 3), per-video `[n/N] title` heartbeat lines, LF-only logs (spinner TTY-gated); probe = `yt-dlp --flat-playlist -J` parsed with jq, new-list diffed against the per-source `--download-archive` BEFORE downloads (exact counts, exact dry-run plans, zero speculative downloads). State machine-owned outside ~/Videos: `$YTSYNC_STATE_DIR/{registry(\x1f-delimited slug⇥type⇥url⇥subdir⇥playlist_title⇥added_ts), archive/<slug>.txt, history.log}`, atomic temp+mv writes; `remove` keeps files AND archive (re-add resumes incrementally); `?v=`+`&list=` URLs download the single video only. Exit codes: 0 incl. no-op/cancel/non-tty-guard; 1 reserved for missing deps, invalid explicit URL, unknown/ambiguous name, wholesale source failure. Notify digest only when new>0 or failed>0 (+ ERR-trap alarm around download passes) via opt-in `lib/notify.sh`. Config scope `ytsync`: `YTSYNC_VIDEOS_DIR` / `YTSYNC_EXTRA_ARGS` (`pos config ytsync`); automation documented as a `pos system schedule` job (`COMMAND=pos media ytsync sync`, `NOTIFY=never`). Deps guards before `-h|--help` with yt-dlp+jq active under `--dry-run` (the preview IS the probe; ffmpeg skipped there). Docs: `tools-docs/ytsync.md` (new dir), POS.md media row + notes, HOWTO.md row, howto/media.md section + troubleshooting, AGENT_Context Common-Tasks row, `bin/pos` EXAMPLES line. Verified: `make gen && make check && make lint` 0 FAIL / 0 WARN; PATH-stub yt-dlp suite (add happy path, incremental 0-new idempotency, playlist NNN numbering, dry-run zero writes, non-tty guard rc0, remove-keeps-archive) with real $HOME byte-untouched via seams.
|
||||
|
||||
@@ -10,19 +10,19 @@
|
||||
|
||||
<!-- GEN:START docmap -->
|
||||
| ## 1. Project Overview | 28–43 |
|
||||
| ## 2. Directory Structure | 44–195 |
|
||||
| ## 3. Installation Flow | 196–247 |
|
||||
| ## 4. The `pos` CLI System | 248–320 |
|
||||
| ## 5. Shared Library — `lib/common.sh` | 321–352 |
|
||||
| ## 6. Docker Compose / ScaleTail | 353–395 |
|
||||
| ## 7. Optional Apps (`apps/`) | 396–425 |
|
||||
| ## 8. Entertainment Module | 426–439 |
|
||||
| ## 9. Systemd Services | 440–451 |
|
||||
| ## 10. Configuration Files | 452–478 |
|
||||
| ## 11. Coding Conventions | 479–511 |
|
||||
| ## 12. Development Workflow | 512–564 |
|
||||
| ## 13. Key File Quick Reference | 565–628 |
|
||||
| ## 14. Common Tasks for Agents | 629–661 |
|
||||
| ## 2. Directory Structure | 44–196 |
|
||||
| ## 3. Installation Flow | 197–248 |
|
||||
| ## 4. The `pos` CLI System | 249–322 |
|
||||
| ## 5. Shared Library — `lib/common.sh` | 323–354 |
|
||||
| ## 6. Docker Compose / ScaleTail | 355–397 |
|
||||
| ## 7. Optional Apps (`apps/`) | 398–427 |
|
||||
| ## 8. Entertainment Module | 428–441 |
|
||||
| ## 9. Systemd Services | 442–453 |
|
||||
| ## 10. Configuration Files | 454–480 |
|
||||
| ## 11. Coding Conventions | 481–513 |
|
||||
| ## 12. Development Workflow | 514–566 |
|
||||
| ## 13. Key File Quick Reference | 567–631 |
|
||||
| ## 14. Common Tasks for Agents | 632–665 |
|
||||
<!-- GEN:END docmap -->
|
||||
|
||||
## 1. Project Overview
|
||||
@@ -80,6 +80,7 @@ Linux_post_install/
|
||||
│ ├── pos-media-mp3 # Download audio as MP3 (yt-dlp)
|
||||
│ ├── pos-media-mp4 # Download video as MP4 (smart/interactive format select)
|
||||
│ ├── pos-media-sync # Incremental Music → USB sync (mp3/mp4, add/update only)
|
||||
│ ├── pos-media-ytsync # Incrementally sync YouTube channels/playlists into ~/Videos
|
||||
│ ├── pos-network-checkport # Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view
|
||||
│ ├── pos-network-download # aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits)
|
||||
│ ├── pos-network-hotspot # Wi-Fi hotspot via create_ap + wihotspot-gui
|
||||
@@ -283,6 +284,7 @@ All non-interactive `pos` commands log output to `~/.local/share/linux_post_inst
|
||||
| media | mp3 | `pos-media-mp3` | Download audio as MP3 (yt-dlp) |
|
||||
| media | mp4 | `pos-media-mp4` | Download video as MP4 (smart/interactive format select) |
|
||||
| media | sync | `pos-media-sync` | Incremental Music → USB sync (mp3/mp4, add/update only) |
|
||||
| media | ytsync | `pos-media-ytsync` | Incrementally sync YouTube channels/playlists into ~/Videos |
|
||||
| network | checkport | `pos-network-checkport` | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view |
|
||||
| network | download | `pos-network-download` | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) |
|
||||
| network | hotspot | `pos-network-hotspot` | Wi-Fi hotspot via create_ap + wihotspot-gui |
|
||||
@@ -583,7 +585,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `features/autostart.sh` | 50 | Boot-time feature (moved from `bin/`, flag-gated service) |
|
||||
| `features/usb-automount.sh` | 138 | USB automount feature (udev rule + flag-gated service) |
|
||||
<!-- GEN:START filetable -->
|
||||
| `bin/pos` | 294 | CLI dispatcher with smart arg matching + logging + category help |
|
||||
| `bin/pos` | 295 | CLI dispatcher with smart arg matching + logging + category help |
|
||||
| `bin/pos-ai-gemini` | 311 | Chat with Google Gemini (ask, chat, models, sessions) |
|
||||
| `bin/pos-communication-matrix-listener` | 568 | Matrix listener: map /command → bash, run them on room messages |
|
||||
| `bin/pos-communication-matrix-sender` | 224 | Send messages to a Matrix room via the client-server API (send, test, login) |
|
||||
@@ -603,6 +605,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-media-mp3` | 86 | Download audio as MP3 (yt-dlp) |
|
||||
| `bin/pos-media-mp4` | 132 | Download video as MP4 (smart/interactive format select) |
|
||||
| `bin/pos-media-sync` | 164 | Incremental Music → USB sync (mp3/mp4, add/update only) |
|
||||
| `bin/pos-media-ytsync` | 1180 | Incrementally sync YouTube channels/playlists into ~/Videos |
|
||||
| `bin/pos-network-checkport` | 496 | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view |
|
||||
| `bin/pos-network-download` | 951 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) |
|
||||
| `bin/pos-network-hotspot` | 93 | Wi-Fi hotspot via create_ap + wihotspot-gui |
|
||||
@@ -610,8 +613,8 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-network-scan` | 272 | Parallel ping sweep of CIDR |
|
||||
| `bin/pos-share-nfs-client` | 138 | Mount NFS shares (ephemeral or persistent systemd mount units) |
|
||||
| `bin/pos-share-nfs-server` | 134 | Manage the NFS kernel server (status, share/unshare exports, enable/disable) |
|
||||
| `bin/pos-share-smb-client` | 217 | Mount SMB/CIFS shares (ephemeral or persistent systemd mount units) |
|
||||
| `bin/pos-share-smb-server` | 253 | Manage the Samba server (status, share/unshare exports, users, enable/disable) |
|
||||
| `bin/pos-share-smb-client` | 440 | Mount SMB/CIFS shares (ephemeral or persistent systemd mount units) |
|
||||
| `bin/pos-share-smb-server` | 311 | Manage the Samba server (status, share/unshare exports, users, enable/disable) |
|
||||
| `bin/pos-share-usb-server` | 218 | USB Redirector server control (--ls, --share; prompts when args omitted) |
|
||||
| `bin/pos-ssh-load-keys` | 31 | Load all SSH keys into the agent |
|
||||
| `bin/pos-system-backup` | 216 | Encrypted (AES-256) folder snapshots (tar + gpg) |
|
||||
@@ -620,7 +623,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-system-schedule` | 81 | Scheduled jobs: run a command on a timer; notify on threshold/change/error/always or silently |
|
||||
| `bin/pos-config` | 80 | Interactive editor for the tools' runtime config (reads # POS_CONFIG: registry) |
|
||||
| `bin/pos-tree` | 112 | Show the pos CLI command tree: categories, commands, and subcommands |
|
||||
| `completions/pos.bash` | 294 | Dynamic bash completion |
|
||||
| `completions/pos.bash` | 298 | Dynamic bash completion |
|
||||
<!-- GEN:END filetable -->
|
||||
| `apps/install.sh` | 171 | App install/uninstall picker/orchestrator |
|
||||
|
||||
@@ -652,6 +655,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| Modify SMB share logic | Edit `bin/pos-share-smb-server` / `bin/pos-share-smb-client` |
|
||||
| Modify scrcpy mirroring logic | Edit `bin/pos-communication-scrcpy` (config scope `scrcpy` via `pos config scrcpy`; `SCRCPY_*` keys in `~/.config/linux_post_install/scrcpy.env`) |
|
||||
| Modify Music→USB sync logic | Edit `bin/pos-media-sync` / shared USB layer `lib/usb-lib.sh` (seams `MEDIA_SYNC_SOURCE`/`MEDIA_SYNC_DEST`/`USB_MOUNT_BASE`/`USB_BYID` in `~/.config/linux_post_install/system.env`) |
|
||||
| Modify YouTube channel sync logic | Edit `bin/pos-media-ytsync` (state in `~/.local/share/linux_post_install/ytsync`; config scope `ytsync` via `pos config ytsync`; research notes `tools-docs/ytsync.md`) |
|
||||
| Modify the scheduler / scheduled jobs | Edit `bin/pos-system-schedule` / `lib/scheduler-lib.sh` (jobs in `~/.config/linux_post_install/schedule.d/`) |
|
||||
| Modify AI/Gemini logic | Edit `bin/pos-ai-gemini` (config scope `ai` via `pos config ai`; `AI_GEMINI_API_KEY`/`AI_GEMINI_MODEL` in `~/.config/linux_post_install/ai.env`) |
|
||||
| Modify UFW/firewall logic | Edit `bin/pos-system-firewall` |
|
||||
|
||||
+1
-1
@@ -12,7 +12,7 @@ authoritative one-line reference (every command + flag), see
|
||||
| `pos ai` | Chat with Google Gemini from CLI or Telegram | [ai](howto/ai.md) |
|
||||
| `pos network` | IP info, hotspot, scan, port check, aria2 download daemon | [network](howto/network.md) |
|
||||
| `pos docker` | Compose services, container dashboards, disposable VMs | [docker](howto/docker.md) |
|
||||
| `pos media` | Download audio/video via yt-dlp | [media](howto/media.md) |
|
||||
| `pos media` | Download audio/video via yt-dlp; incremental YouTube channel sync | [media](howto/media.md) |
|
||||
| `pos system` | Backups, firewall, health dashboard | [system](howto/system.md) |
|
||||
| `pos system schedule` | Scheduled jobs: run a command on a timer, notify on threshold/change/error or silently | [schedule](howto/schedule.md) |
|
||||
| `pos ssh` | Load keys into the agent | [ssh](howto/ssh.md) |
|
||||
|
||||
+11
@@ -210,6 +210,17 @@ The standalone `vbox` command still works and forwards to `pos docker vbox` (see
|
||||
| `pos media mp3 <url>` | `bin/pos-media-mp3` | Download audio as MP3 via yt-dlp, with thumbnail + metadata | Output to `~/Music/%(title)s.%(ext)s`, `--audio-quality 0` |
|
||||
| `pos media mp4 <url>` | `bin/pos-media-mp4` | Download video via yt-dlp with **interactive format selection** | Lists formats (`yt-dlp -F`), asks for a format ID, saves to `~/Videos/` |
|
||||
| `pos media sync [--mp3\|--mp4]` | `bin/pos-media-sync` | Incremental Music → USB sync (add/update only — never deletes) | Copies mp3/mp4 from `$HOME/Music` (or `--source <dir>`) into `<usb>/Music/`, preserving the tree; missing or changed (size/mtime) files are copied, identical ones skipped. Same USB detection as `pos system backup` (lsblk TRAN + lsusb/by-id, mount offer for unmounted sticks, multi-stick picker). `--mp3`/`--mp4` filter by extension, neither = both; `--dry-run` previews. Config: `MEDIA_SYNC_SOURCE`, `MEDIA_SYNC_DEST`, shared `USB_MOUNT_BASE`/`USB_BYID` from `~/.config/linux_post_install/system.env`. Result notified via `lib/notify.sh` |
|
||||
| `pos media ytsync [add\|sync\|list\|remove]` | `bin/pos-media-ytsync` | Incremental YouTube channel/playlist sync — first run asks for a URL (bare invocation = interactive menu; empty state goes straight to the prompt), repeat runs fetch only new videos | One yt-dlp call per new video (`bestvideo*+bestaudio/best` → MP4, metadata/chapters/thumbnail embedded, `--no-overwrites`, `--windows-filenames --trim-filenames 120`); per-source `--download-archive` (`~/.local/share/linux_post_install/ytsync/archive/<slug>.txt`) makes runs crash-safe and idempotent; registry tracks slug/type/url/subdir. Verbs never prompt (scheduler/timer safe); non-tty interactive entry prints a guard line and exits 0. `--dry-run` probes + plans with zero writes. Notify digest only when new>0 or failed>0 via `lib/notify.sh`. Config: `YTSYNC_VIDEOS_DIR`, `YTSYNC_EXTRA_ARGS` via `pos config ytsync`; automate with `pos system schedule` (`COMMAND=pos media ytsync sync`, `NOTIFY=never`) |
|
||||
|
||||
A watch link carrying **both** `?v=` and `&list=` downloads only that single video
|
||||
(`--no-playlist`) — nobody accidentally backfills a 500-video playlist from a watch
|
||||
link; a pure playlist link becomes a tracked playlist source with numbered
|
||||
`<NNN> - <title>.mp4` files. `pos media ytsync remove <name>` stops tracking but
|
||||
keeps the downloaded files AND the archive — re-adding the same source later
|
||||
resumes incrementally instead of re-downloading. Members-only/age-gated videos are
|
||||
reported as "N videos require sign-in — skipped" (escape hatch:
|
||||
`YTSYNC_EXTRA_ARGS="--cookies …"` in `ytsync.env`). Research details:
|
||||
`tools-docs/ytsync.md`.
|
||||
|
||||
### system
|
||||
|
||||
|
||||
+92
-2
@@ -1,13 +1,15 @@
|
||||
# How-To: `pos media`
|
||||
|
||||
Download audio and video from the web via `yt-dlp`, and sync your library to a
|
||||
USB stick. Tools: `mp3`, `mp4`, `sync`.
|
||||
Download audio and video from the web via `yt-dlp`, sync your library to a
|
||||
USB stick, and keep YouTube channels incrementally up to date.
|
||||
Tools: `mp3`, `mp4`, `sync`, `ytsync`.
|
||||
|
||||
| Tool | What it does |
|
||||
|------|--------------|
|
||||
| `pos media mp3` | Download audio, convert to MP3 |
|
||||
| `pos media mp4` | Download video with smart/interactive format selection |
|
||||
| `pos media sync` | Incrementally copy `~/Music` onto a USB stick (mp3/mp4) |
|
||||
| `pos media ytsync` | Track YouTube channels/playlists and download only new videos into `~/Videos` |
|
||||
|
||||
Requires `yt-dlp` and `ffmpeg` (`sudo apt install yt-dlp ffmpeg`); the tools
|
||||
fail with a clean error message instead of a raw `command not found` if either
|
||||
@@ -147,6 +149,85 @@ Config (all in `~/.config/linux_post_install/system.env` or exported):
|
||||
|
||||
---
|
||||
|
||||
## `pos media ytsync` — incremental YouTube channel sync
|
||||
|
||||
```bash
|
||||
pos media ytsync # interactive
|
||||
pos media ytsync add https://youtube.com/@SomeChannel
|
||||
pos media ytsync sync # fetch new videos from every source
|
||||
pos media ytsync list
|
||||
pos media ytsync remove "Some Channel"
|
||||
```
|
||||
|
||||
The first run asks for a channel or playlist URL, shows what was resolved
|
||||
(`Resolved : Linus Tech Tips (channel · 2140 videos)` + the target library path),
|
||||
and downloads everything. Every later run probes the same URL, diffs it against a
|
||||
per-source download archive, and fetches **only new videos** — one yt-dlp call per
|
||||
video (`bestvideo*+bestaudio/best` merged to MP4 with metadata, chapters and the
|
||||
thumbnail embedded; existing files are never overwritten).
|
||||
|
||||
Where files land:
|
||||
|
||||
```
|
||||
~/Videos/<channel>/<title>.mp4 # channels & single videos
|
||||
~/Videos/<channel>/<playlist>/<NNN> - <title>.mp4 # playlist sources (playlist order)
|
||||
```
|
||||
|
||||
| Command | Meaning |
|
||||
|---------|---------|
|
||||
| `add [url]` | Register a source + first download (asks for the URL if omitted) |
|
||||
| `sync [name]` | Incremental pass — all tracked sources, or one by slug/display name |
|
||||
| `list` | Tracked sources table: type, archived count, last sync date, destination |
|
||||
| `remove <name>` | Stop tracking. Downloaded files AND the archive are kept |
|
||||
|
||||
Notes:
|
||||
|
||||
- A watch link with **both** `?v=` and `&list=` downloads only that single video,
|
||||
never the whole playlist.
|
||||
- Retitled/renamed videos keep their local filenames (the archive is keyed by video
|
||||
id); inserting a video mid-playlist shifts future numbering only.
|
||||
- Members-only / age-gated videos are skipped with a count ("N videos require
|
||||
sign-in"). Escape hatch: put `YTSYNC_EXTRA_ARGS=--cookies-from-browser firefox`
|
||||
(or `--cookies <file>`) into the config file below.
|
||||
- State lives outside `~/Videos`, in `~/.local/share/linux_post_install/ytsync/`
|
||||
(registry, per-source archives, run history) — the media tree stays pure media.
|
||||
|
||||
**Recipe: daily automation** — no built-in timers; use the shared scheduler:
|
||||
|
||||
```bash
|
||||
pos system schedule config # name: ytsync
|
||||
# INTERVAL=daily (or hourly / weekly / OnCalendar=…)
|
||||
# NOTIFY=never # ytsync sends its own digest; don't double-notify
|
||||
# MSG="ytSync"
|
||||
# COMMAND=pos media ytsync sync
|
||||
pos system schedule enable ytsync
|
||||
pos system schedule run ytsync # test once, right now
|
||||
```
|
||||
|
||||
ytsync never prompts on `sync`, so the job is timer-safe by construction. It
|
||||
notifies via `lib/notify.sh` only when something happened (new videos or failures);
|
||||
a scheduled no-op run stays silent — hence `NOTIFY=never` on the job to avoid
|
||||
double alerts.
|
||||
|
||||
**Previewing:** `--dry-run` works on `add` and `sync` — real probe + plan block
|
||||
(`New : 12 would be downloaded (800 already present)` + example filenames), zero
|
||||
writes anywhere:
|
||||
|
||||
```bash
|
||||
pos media ytsync sync --dry-run
|
||||
pos media ytsync add https://youtube.com/@SomeChannel --dry-run
|
||||
```
|
||||
|
||||
Config (`~/.config/linux_post_install/ytsync.env`, chmod 600 — materialize/edit via
|
||||
`pos config ytsync`; exported environment wins):
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
| `YTSYNC_VIDEOS_DIR` | `$HOME/Videos` | Videos root for synced sources |
|
||||
| `YTSYNC_EXTRA_ARGS` | *(empty)* | Extra yt-dlp flags appended to every download call (cookies, format overrides, …) |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Format list is empty / download fails → the site or age-gate requires
|
||||
@@ -157,6 +238,15 @@ Config (all in `~/.config/linux_post_install/system.env` or exported):
|
||||
it falls back to the uploader name in the artist slot.
|
||||
- Very large downloads: ensure free space; files land in `~/Music`/`~/Videos`
|
||||
(or your `-o` directory).
|
||||
- ytsync: "could not reach YouTube" at add → connectivity/DNS; the interactive
|
||||
flow re-prompts up to 3×. "source not found or private" → wrong handle or a
|
||||
deleted/private source.
|
||||
- ytsync: many "exists, kept" warnings after a YouTube-side change → the
|
||||
channel's internal id changed, so re-adding computed a new archive. Rename the
|
||||
old archive file (`~/.local/share/linux_post_install/ytsync/archive/<old>.txt`)
|
||||
to the new slug before syncing (see `tools-docs/ytsync.md`).
|
||||
- ytsync: "disk full — stopping … mid-run" → free space on the videos volume and
|
||||
run `pos media ytsync sync` again; unfetched videos simply stay "new".
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -153,6 +153,7 @@ EXAMPLES
|
||||
|
||||
pos media mp3 <url> Download audio as MP3
|
||||
pos media mp4 <url> Download video as MP4
|
||||
pos media ytsync sync Incremental YouTube channel sync
|
||||
|
||||
pos system firewall Interactive UFW manager
|
||||
pos system backup /srv/project Encrypted (AES-256) folder snapshot
|
||||
|
||||
Executable
+1180
File diff suppressed because it is too large
Load Diff
@@ -12,6 +12,7 @@ _pos_flags[entertainment-send]="--print --markdown"
|
||||
_pos_flags[media-mp3]="--output --no-playlist --cookies --by-artist --dry-run"
|
||||
_pos_flags[media-mp4]="--format --best --worst --output --no-playlist --cookies --dry-run"
|
||||
_pos_flags[media-sync]="--mp3 --mp4 --source --dry-run"
|
||||
_pos_flags[media-ytsync]="--dry-run"
|
||||
_pos_flags[network-checkport]="--tcp --udp --ping --no-banner --versions --timeout"
|
||||
_pos_flags[network-download]="--dir --out --split --seed --force --upload --gid --tmux"
|
||||
_pos_flags[network-hotspot]="--foreground"
|
||||
@@ -28,11 +29,14 @@ _pos_subcmds[communication-scrcpy]="devices record tcpip connect push pull scree
|
||||
_pos_subcmds[communication-telegram-sender]="send test"
|
||||
_pos_subcmds[docker-compose]="ls installed up down restart logs update config"
|
||||
_pos_subcmds[docker-vbox]="create enter stop start rm ls"
|
||||
_pos_subcmds[media-ytsync]="add sync list remove"
|
||||
_pos_subcmds[network-download]="start stop status add torrent metalink list info files peers pause resume remove purge move limit set watch restart retry replace"
|
||||
_pos_subcmds[share-smb-client]="mount unmount list persist unpersist"
|
||||
_pos_subcmds[share-smb-server]="status share unshare list adduser deluser reload enable disable"
|
||||
_pos_subcmds[system-schedule]="run list config enable disable status migrate"
|
||||
# GEN:END possubcmds
|
||||
# GEN:START posconfigscopes
|
||||
declare -a _pos_config_scopes=(ai compose entertainment matrix notify scrcpy system telegram)
|
||||
declare -a _pos_config_scopes=(ai compose entertainment matrix notify scrcpy system telegram ytsync)
|
||||
# GEN:END posconfigscopes
|
||||
|
||||
_pos() {
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
# `pos media ytsync` — implementation research notes
|
||||
|
||||
Internal reference for maintainers touching `bin/pos-media-ytsync`. The user-facing
|
||||
docs live in `DOC/POS.md → media` and `DOC/howto/media.md`; the behavioral contract
|
||||
is `reportAgents/2026-08-22-designer-ytsync.md` + the Architect decisions D1–D9 in
|
||||
`reportAgents/2026-08-22-architect-ytsync.md`.
|
||||
|
||||
## Probe mechanics
|
||||
|
||||
One fast call per source per run:
|
||||
|
||||
```
|
||||
yt-dlp --flat-playlist -J --no-warnings -- <url>
|
||||
```
|
||||
|
||||
- `--flat-playlist` keeps entries as stubs, so cost is 1–2 HTTP round trips
|
||||
regardless of library size (seconds even for 2000-video channels).
|
||||
- `-J` dumps a single JSON object. Parsed with `jq`:
|
||||
- display name: `.channel // .uploader // .uploader_id // .title`
|
||||
- owner (for playlist subdirs): `.channel // .uploader // .uploader_id` — no title
|
||||
fallback (avoids `Playlist Title/Playlist Title` doubling when a playlist probe
|
||||
has no channel fields)
|
||||
- key (slug source): `.uploader_id // .channel_id // .id`
|
||||
- entries: `.entries[] | (.id) + "\x1f" + (.title)`; a `?v=` URL returns a bare
|
||||
video object with **no** `.entries`, handled as a one-entry list.
|
||||
- The new-list is computed BEFORE any download by diffing entry ids against
|
||||
`archive/<slug>.txt` — exact `[n/N]` counts, exact dry-run plans, zero speculative
|
||||
downloads.
|
||||
- Sign-in-skipped bucket: entries with an empty id OR titles starting with
|
||||
`[Private` / `[Deleted` / `[Unavailable`. Reported as "N videos require sign-in —
|
||||
skipped"; never counted as failures.
|
||||
|
||||
### Why probes don't use spawn()
|
||||
|
||||
`lib/common.sh spawn()` **exits the process on failure** and captures output until
|
||||
completion. ytsync needs recoverable probes (interactive add re-prompts ≤3×; sync
|
||||
must continue other sources after one bad probe) and never wraps the download batch
|
||||
(hours of silence behind a spinner). `_probe_run` mirrors spawn's OK/FAIL line UX but
|
||||
returns rc, and its braille spinner frames are emitted only on a TTY so no `\r`
|
||||
bytes ever reach the dispatcher's tee'd logs.
|
||||
|
||||
## Classification (URL shape, deterministic)
|
||||
|
||||
| URL contains | Type | Behavior |
|
||||
|---|---|---|
|
||||
| `list=` without `v=` | `playlist` | tracked source; files land as `<videos>/<owner>/<playlist>/<NNN> - <title>.<ext>` |
|
||||
| `v=` present (with or without `list=`) | `video` | single video, downloaded with `--no-playlist`; nobody backfills a 500-video playlist by pasting a watch link |
|
||||
| neither | `channel` | flat source; `<videos>/<channel>/<title>.<ext>` |
|
||||
|
||||
## Download invocation (one yt-dlp call PER NEW VIDEO)
|
||||
|
||||
```
|
||||
yt-dlp -f "bestvideo*+bestaudio/best"
|
||||
--merge-output-format mp4 # parity with pos media mp4
|
||||
--embed-metadata --embed-chapters --embed-thumbnail
|
||||
--no-overwrites # parity; collisions become "exists, kept" warnings
|
||||
--download-archive <state>/archive/<slug>.txt # crash-safe per-video recording
|
||||
--windows-filenames # USB/Samba/TV-safe names
|
||||
--trim-filenames 120 # headroom for the NNN prefix under 255-byte limits
|
||||
--retries 3 --fragment-retries 3
|
||||
[--no-playlist] # only for type=video sources
|
||||
${YTSYNC_EXTRA_ARGS} # appended last — user override hatch
|
||||
-o "<template>" "https://www.youtube.com/watch?v=<id>"
|
||||
```
|
||||
|
||||
- Templates are COMPUTED from stored registry fields (`subdir`), never from yt-dlp
|
||||
placeholders like `%(playlist_title)s` — a stored subdir cannot be NA and cannot
|
||||
drift mid-library.
|
||||
- Playlist template gets a LITERAL zero-padded index injected by the tool (from the
|
||||
probe's entry position), because standalone watch URLs have no live
|
||||
`%(playlist_index)s`. Caveat: inserting a video mid-playlist shifts FUTURE
|
||||
numbering; existing files are never renamed.
|
||||
- Deliberate divergences from mp4: no `--embed-subs --sub-langs all` (library bloat;
|
||||
re-add per-source via `YTSYNC_EXTRA_ARGS`, appended-last wins); `--quiet
|
||||
--no-warnings` plus `--progress` only on a TTY (yt-dlp auto-simplifies progress
|
||||
when piped, keeping `\r` out of logs).
|
||||
- Sequential downloads; batching is the listed future optimization (accepted v1
|
||||
cost: one extra extraction round trip per video).
|
||||
|
||||
## State layout (machine-owned, outside ~/Videos)
|
||||
|
||||
```
|
||||
${YTSYNC_STATE_DIR:-~/.local/share/linux_post_install/ytsync}/
|
||||
├── registry # \x1f-delimited: slug⇥type⇥url⇥subdir⇥playlist_title⇥added_ts
|
||||
├── archive/<slug>.txt # native yt-dlp archive format ("<extractor> <id>"), one per source
|
||||
└── history.log # append-only: "<YYYY-MM-DD HH:MM> · <name> · N new · M skipped · K failed"
|
||||
```
|
||||
|
||||
- `slug`: `[a-z0-9][a-z0-9_-]*`, derived from probed uploader_id/handle; numeric
|
||||
`-2` suffix on collision, checking registry slugs only.
|
||||
- Registry/archive writes are atomic (mktemp inside the state dir + mv).
|
||||
- `remove` drops only the registry line. Keeping the orphaned archive makes a future
|
||||
re-add of the same source an incremental resume instead of a full re-download that
|
||||
would collide with existing files under `--no-overwrites`.
|
||||
- History grammar note: a wholesale probe failure appends `<date> · <name> · FAILED
|
||||
(probe)` instead of the numeric triple — the pass did not complete, and faking
|
||||
zeros would hide it from `grep`.
|
||||
|
||||
## Edge cases
|
||||
|
||||
- **Mid-playlist inserts** shift future numbering only (see above).
|
||||
- **Title renames on YouTube** never rename local files — the archive is keyed by
|
||||
video id; local files are immutable once written.
|
||||
- **Same-title collision** (`--no-overwrites` refusal): stderr is matched for
|
||||
"already been downloaded" → `[!] exists, kept: <file>`, counted into the
|
||||
"already present" tally of the summary (the fixed summary grammar has three
|
||||
buckets; tools-docs records this folding).
|
||||
- **Disk full (ENOSPC)**: stderr matched for "No space left on device"/Errno 28 →
|
||||
prominent warning, that source stops mid-run (unfetched videos simply stay "new"
|
||||
next run since the archive was untouched), other channels continue.
|
||||
- **Per-video generic failure**: `[!] unavailable: <title>` + first 3 stderr lines,
|
||||
continue, counted as failed. Per-video failures never flip the exit code; rc 1 is
|
||||
reserved for missing deps, invalid explicit URL, unknown/ambiguous `<name>`, or
|
||||
≥1 requested source failing wholesale during sync.
|
||||
- **Slug stability**: the slug derives from the probed `uploader_id`. If YouTube
|
||||
ever changes that id for a channel, a re-add computes a different slug whose
|
||||
archive starts empty → existing files would be "re-downloaded", hitting
|
||||
`--no-overwrites` and spamming `exists, kept` warnings. Manual mitigation if it
|
||||
ever bites: rename the old `archive/<old-slug>.txt` to the new slug before
|
||||
syncing. Accepted v1 limitation (Architect §Remaining uncertainty #4).
|
||||
|
||||
## Testing seams
|
||||
|
||||
- `YTSYNC_STATE_DIR`, `YTSYNC_VIDEOS_DIR` — every written path honors them
|
||||
(written `VAR="${VAR:-default}"`). Fake `yt-dlp` arrives via stub PATH emitting
|
||||
canned `-J` JSON; fake notify via a stubbed sender appending to `sends.log`.
|
||||
- Interactive reads all come from `/dev/tty` (NOT stdin), so the tool is NOT in
|
||||
`bin/pos`'s INTERACTIVE_CMDS and dispatched runs keep full tee logging; non-tty
|
||||
interactive entry points print the guard line and exit 0.
|
||||
Reference in New Issue
Block a user