diff --git a/AGENT_TODO.md b/AGENT_TODO.md index 9288c47..54a78e1 100644 --- a/AGENT_TODO.md +++ b/AGENT_TODO.md @@ -42,6 +42,8 @@ summary (newest last). ## Done +- **2026-09-09** — Unified YouTube tools into `pos media yt` + new `subtitles` (Architect POS--9). New `bin/pos-media-yt` dispatcher (mp3/mp4/grab/ytsync/subtitles) + `bin/pos-media-yt-{mp3,mp4,grab,subtitles,ytsync}`; the ytsync file is a forwarder to the existing `pos media ytsync`; legacy `bin/pos-media-{mp3,mp4,grab}` became thin forwarders to the `yt` forms. New `lib/yt-lib.sh` (deps/URL-validation/echo/classify helpers; `classify_url` migrated from grab, `yt_validate_url` is a return-1 checker — never exits, so callers can prefix errors). `bin/pos` INTERACTIVE_CMDS += `media-yt-mp4` (interactive format pick reads stdin). `pos-media-yt-subtitles` extracts captions via `--write-subs --write-auto-subs --sub-langs best`, `--lang en,ar` (one `--sub-langs` arg), `srt|vtt|txt` (txt = srt→txt conversion stripping timestamps/HTML), `--auto-only`, `--list-subs` probe, `--output`, no-ffmpeg dep (yt-dlp only; dry-run skips deps entirely). Docs: DOC/POS.md media section rewritten (yt group + forwarder rows), DOC/howto/media.md `yt` commands + subtitles section, AGENT_Context hand-maintained `lib/yt-lib.sh` row, tests/README row. New `tests/t-pos-media-yt.sh` (72 checks: dispatcher/forwarder resolution, full `pos media mp3` dispatch chain, yt-lib helpers, per-tool flags/dry-run/`YT_OUT_DIR` seam/`GRAB_DEFAULT` config, 3 mandated negative controls — unsafe-URL no-expansion, `--lang en,ar` single arg, txt timestamp-stripping, unavailable-subs detection). `tests/t-config-precedence.sh` Part D config-consumer list updated `pos-media-grab` → `pos-media-yt-grab`. Verified: `bash -n` all; `make gen` ×2 byte-idempotent; `make check` OK; `make lint` 0 FAIL / 0 WARN; `make test` 21 files / 533 checks / 0 fail / 0 skip; `git diff --check` clean; smokes — `pos media yt --help`, `yt mp3/mp4/subtitles --help`, `yt ytsync --help` (reaches `pos media ytsync`), `pos media mp3 --help` forwarder, `pos tree` shows the `yt` subtree (with repo-first PATH; system `/usr/local/bin` has a stale pre-POS--9 install that shadows it otherwise). + - **2026-09-08** — `install.sh` version gate (Architect→Builder): skip+abort when installed version == current version, `--force` to bypass, version scheme `0.0c` (auto-bumps per commit). `install_version()` derives `0.0c$(git rev-list --count HEAD)`; empty when `.git` absent → gate skipped (silently); `INSTALL_VERSION_OVERRIDE` env var (presence-check) = test seam. Gate after arg-parse, before phases, numeric comparison (strip `0.0c`, `-eq`); `log "Already installed ($CURRENT_VERSION). Use --force to re-install."` / `--dry-run` → `(dry-run) Would skip install: already at version $CURRENT_VERSION`, both exit 0. `FORCE=0` init, `--force` parse + usage. `flag_set installed_version "$CURRENT_VERSION"` after "Bootstrap complete" banner (only when DRY_RUN≠1 and version non-empty; even under --force). New `tests/t-install-version.sh` (21 checks / 9 cases). Docs: README/SCRIPTS/AGENT_Context (flags, flow, line count 248→301, tests/README row). Verified: `bash -n` clean; `make gen` idempotent; `make check` OK; `make lint` 0 FAIL / 0 WARN; `make test` suite green. diff --git a/DOC/AGENT_Context_Project.md b/DOC/AGENT_Context_Project.md index 3d07d23..dcb5884 100644 --- a/DOC/AGENT_Context_Project.md +++ b/DOC/AGENT_Context_Project.md @@ -10,19 +10,19 @@ | ## 1. Project Overview | 28–43 | -| ## 2. Directory Structure | 44–210 | -| ## 3. Installation Flow | 211–269 | -| ## 4. The `pos` CLI System | 270–351 | -| ## 5. Shared Library — `lib/common.sh` | 352–383 | -| ## 6. Docker Compose / ScaleTail | 384–426 | -| ## 7. Optional Apps (`apps/`) | 427–456 | -| ## 8. Entertainment Module | 457–470 | -| ## 9. Systemd Services | 471–482 | -| ## 10. Configuration Files | 483–509 | -| ## 11. Coding Conventions | 510–542 | -| ## 12. Development Workflow | 543–595 | -| ## 13. Key File Quick Reference | 596–671 | -| ## 14. Common Tasks for Agents | 672–705 | +| ## 2. Directory Structure | 44–216 | +| ## 3. Installation Flow | 217–275 | +| ## 4. The `pos` CLI System | 276–363 | +| ## 5. Shared Library — `lib/common.sh` | 364–395 | +| ## 6. Docker Compose / ScaleTail | 396–438 | +| ## 7. Optional Apps (`apps/`) | 439–468 | +| ## 8. Entertainment Module | 469–482 | +| ## 9. Systemd Services | 483–494 | +| ## 10. Configuration Files | 495–521 | +| ## 11. Coding Conventions | 522–554 | +| ## 12. Development Workflow | 555–607 | +| ## 13. Key File Quick Reference | 608–690 | +| ## 14. Common Tasks for Agents | 691–724 | ## 1. Project Overview @@ -90,7 +90,13 @@ Linux_post_install/ │ ├── pos-media-mp4 # Download video as MP4 (smart/interactive format select) │ ├── pos-media-sync # Incremental Music → USB sync (mp3/mp4, add/update only) │ │ [deps: lsblk jq] +│ ├── pos-media-yt-grab # Auto-download URL as audio or video (classify + route) +│ ├── pos-media-yt-mp3 # Download audio as MP3 (yt-dlp) +│ ├── pos-media-yt-mp4 # Download video as MP4 (smart/interactive format select) +│ ├── pos-media-yt-subtitles # Extract subtitles/captions from a URL (yt-dlp) +│ ├── pos-media-yt-ytsync # Incrementally sync YouTube channels/playlists into ~/Videos │ ├── pos-media-ytsync # Incrementally sync YouTube channels/playlists into ~/Videos +│ ├── pos-media-yt # YouTube download tools (mp3/mp4/grab/ytsync/subtitles) │ ├── pos-network-checkport # Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view │ ├── pos-network-download # aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) │ │ [deps: aria2c jq curl] @@ -311,7 +317,13 @@ All non-interactive `pos` commands log output to `~/.local/share/linux_post_inst | media | mp3 | `pos-media-mp3` | Download audio as MP3 (yt-dlp) | | | | media | mp4 | `pos-media-mp4` | Download video as MP4 (smart/interactive format select) | | | | media | sync | `pos-media-sync` | Incremental Music → USB sync (mp3/mp4, add/update only) | lsblk jq | pos media sync --mp3 → Sync only MP3 files to USB · pos media sync --mp4 --dry-run → Preview MP4 sync without copying | +| media | yt-grab | `pos-media-yt-grab` | Auto-download URL as audio or video (classify + route) | | | +| media | yt-mp3 | `pos-media-yt-mp3` | Download audio as MP3 (yt-dlp) | | | +| media | yt-mp4 | `pos-media-yt-mp4` | Download video as MP4 (smart/interactive format select) | | | +| media | yt-subtitles | `pos-media-yt-subtitles` | Extract subtitles/captions from a URL (yt-dlp) | | | +| media | yt-ytsync | `pos-media-yt-ytsync` | Incrementally sync YouTube channels/playlists into ~/Videos | | | | media | ytsync | `pos-media-ytsync` | Incrementally sync YouTube channels/playlists into ~/Videos | | | +| media | yt | `pos-media-yt` | YouTube download tools (mp3/mp4/grab/ytsync/subtitles) | | | | network | checkport | `pos-network-checkport` | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view | | | | network | download | `pos-network-download` | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) | aria2c jq curl | pos network download add https://example.com/file.zip → Enqueue an HTTP download (auto-starts daemon) · pos network download status → Daemon health + global transfer stats · pos network download watch → Live progress view | | network | hotspot | `pos-network-hotspot` | Wi-Fi hotspot via create_ap + wihotspot-gui | | | @@ -376,7 +388,7 @@ source "$(dirname "$0")/../lib/common.sh" **Scripts that do NOT source common.sh** (self-contained): -`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`. +`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-media-grab`, `pos-media-mp3`, `pos-media-mp4`, `pos-media-yt-ytsync`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`. --- @@ -611,6 +623,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | `lib/share-lib.sh` | 318 | Domain layer for the share suite (usbsrv/smbclient record parsers, folder+mountpoint candidates, remote listings, service/firewall advisories; EOF-safe) + compat shims to `lib/menu-lib.sh` — used by all five `pos share *` tools | | `lib/menu-lib.sh` | 362 | Category-neutral interactive menu primitives (`menu_guard` tty guard, `menu_run` looping boxed menu, `menu_pick` type-to-filter picker, `menu_ask_value` prompt-with-default via raw-mode bracketed-paste-safe `menu_read_value`; stderr render, fail-closed on non-tty/EOF) — sourced by `share-lib.sh`, open to any category | | `lib/registry.sh` | 199 | Shared query API for POS tool metadata headers (`# POS_*:`) — `reg_scan`/`reg_list`/`reg_lookup`/`reg_each`/config scope helpers; used by `pos-tree` and `gen-docs.sh` | +| `lib/yt-lib.sh` | 50 | Shared YouTube helpers for `pos media yt *` (`yt_check_deps`, `yt_validate_url`, `yt_echo_cmd`, `classify_url`) — sourced by `yt-mp3`/`yt-mp4`/`yt-grab`/`yt-subtitles` | | `bin/flag-reader` | 58 | Inspect flags (list/status/`--raw`) | | `bin/flag-set` | 21 | Set a flag (optionally with a value) | | `bin/flag-clear` | 21 | Unset a flag | @@ -639,11 +652,17 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | `bin/pos-entertainment-enable` | 49 | Enable an auto-trigger for a plugin on a schedule | | `bin/pos-entertainment-send` | 95 | Run a public-API plugin and send its output via the configured notify platforms | | `bin/pos-entertainment-status` | 62 | Show enabled plugins and scheduler state | -| `bin/pos-media-grab` | 219 | Auto-download URL as audio or video (classify + route) | -| `bin/pos-media-mp3` | 86 | Download audio as MP3 (yt-dlp) | -| `bin/pos-media-mp4` | 132 | Download video as MP4 (smart/interactive format select) | +| `bin/pos-media-grab` | 5 | Auto-download URL as audio or video (classify + route) | +| `bin/pos-media-mp3` | 5 | Download audio as MP3 (yt-dlp) | +| `bin/pos-media-mp4` | 5 | Download video as MP4 (smart/interactive format select) | | `bin/pos-media-sync` | 219 | Incremental Music → USB sync (mp3/mp4, add/update only) | +| `bin/pos-media-yt-grab` | 208 | Auto-download URL as audio or video (classify + route) | +| `bin/pos-media-yt-mp3` | 86 | Download audio as MP3 (yt-dlp) | +| `bin/pos-media-yt-mp4` | 132 | Download video as MP4 (smart/interactive format select) | +| `bin/pos-media-yt-subtitles` | 169 | Extract subtitles/captions from a URL (yt-dlp) | +| `bin/pos-media-yt-ytsync` | 5 | Incrementally sync YouTube channels/playlists into ~/Videos | | `bin/pos-media-ytsync` | 1213 | Incrementally sync YouTube channels/playlists into ~/Videos | +| `bin/pos-media-yt` | 33 | YouTube download tools (mp3/mp4/grab/ytsync/subtitles) | | `bin/pos-network-checkport` | 498 | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view | | `bin/pos-network-download` | 1113 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) | | `bin/pos-network-hotspot` | 93 | Wi-Fi hotspot via create_ap + wihotspot-gui | @@ -663,7 +682,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | `bin/pos-ai` | 714 | AI assistant: ask, chat, sessions, capture, models, providers | | `bin/pos-config` | 80 | Interactive editor for the tools' runtime config (reads # POS_CONFIG: registry) | | `bin/pos-tree` | 118 | Show the pos CLI command tree: categories, commands, and subcommands | -| `completions/pos.bash` | 314 | Dynamic bash completion | +| `completions/pos.bash` | 316 | Dynamic bash completion | | `apps/install.sh` | 171 | App install/uninstall picker/orchestrator | diff --git a/DOC/POS.md b/DOC/POS.md index 98b3ec0..116703a 100644 --- a/DOC/POS.md +++ b/DOC/POS.md @@ -263,9 +263,15 @@ The standalone `vbox` command still works and forwards to `pos docker vbox` (see | Command | File | Purpose | Configuration | |---------|------|---------|---------------| -| `pos media mp3 ` | `bin/pos-media-mp3` | Download audio as MP3 via yt-dlp, with thumbnail + metadata | Output to `~/Music/%(title)s.%(ext)s`, `--audio-quality 0` | -| `pos media mp4 ` | `bin/pos-media-mp4` | Download video via yt-dlp with **interactive format selection** | Lists formats (`yt-dlp -F`), asks for a format ID, saves to `~/Videos/` | -| `pos media grab ` | `bin/pos-media-grab` | Auto-download a URL as audio or video (classify + route to mp3/mp4) | Domain-based classification (YouTube Music/SoundCloud/Bandcamp → audio; YouTube/Vimeo/Twitch → video); `--audio`/`--video` force the mode; `--best` default for video (non-interactive); prints a clean summary (🎵/🎬 title, path, size). Config: `GRAB_DEFAULT` (`pos config grab`, default `video`) for unknown domains | +| `pos media yt` | `bin/pos-media-yt` | YouTube download tools dispatcher (mp3/mp4/grab/ytsync/subtitles); bare invocation prints help listing subcommands | Subcommands resolved via `bin/pos-media-yt-` files | +| `pos media yt mp3 ` | `bin/pos-media-yt-mp3` | Download audio as MP3 via yt-dlp, with thumbnail + metadata | Output to `~/Music/%(title)s.%(ext)s`, `--audio-quality 0`; `--by-artist` organizes as `//.mp3`. Env seam: `YT_OUT_DIR` overrides the default output dir | +| `pos media yt mp4 <url>` | `bin/pos-media-yt-mp4` | Download video via yt-dlp with **interactive format selection** | Lists formats (`yt-dlp -F`), asks for a format ID, saves to `~/Videos/`; `-f/--best/--worst` skip the prompt. Env seam: `YT_OUT_DIR` overrides the default output dir. Reads stdin (interactive format pick) → in `INTERACTIVE_CMDS` | +| `pos media yt grab <url>` | `bin/pos-media-yt-grab` | Auto-download a URL as audio or video (classify + route to yt-mp3/yt-mp4) | Domain-based classification (YouTube Music/SoundCloud/Bandcamp → audio; YouTube/Vimeo/Twitch → video); `--audio`/`--video` force the mode; `--best` default for video (non-interactive); prints a clean summary (🎵/🎬 title, path, size). Config: `GRAB_DEFAULT` (`pos config grab`, default `video`) for unknown domains | +| `pos media yt ytsync [add\|sync\|list\|remove]` | `bin/pos-media-yt-ytsync` | Thin forwarder → `pos media ytsync` (incremental YouTube channel/playlist sync into `~/Videos`) | See `bin/pos-media-ytsync` below | +| `pos media yt subtitles <url>` | `bin/pos-media-yt-subtitles` | Extract subtitles/captions from a URL (yt-dlp) | Default `--write-subs --write-auto-subs --sub-langs best`; `--lang en,ar` (comma = single `--sub-langs` arg); `--format srt` (default) / `vtt` / `txt` (srt→txt post-conversion); `--auto-only` drops manual subs; `--list-subs` probe; output to current dir (`-o ./%(title)s.%(sub_lang)s.%(ext)s`), `--output <dir>` overrides. Non-interactive; needs only yt-dlp (no ffmpeg) | +| `pos media mp3 <url>` | `bin/pos-media-mp3` | **Forwarder** → `pos media yt mp3` (backward-compat alias) | Legacy spelling still works | +| `pos media mp4 <url>` | `bin/pos-media-mp4` | **Forwarder** → `pos media yt mp4` (backward-compat alias) | Legacy spelling still works | +| `pos media grab <url>` | `bin/pos-media-grab` | **Forwarder** → `pos media yt grab` (backward-compat alias) | Legacy spelling still works | | `pos media sync [--mp3\|--mp4]` | `bin/pos-media-sync` | Incremental Music → USB sync (add/update only — never deletes) | Copies mp3/mp4 from `$HOME/Music` (or `--source <dir>`) into `<usb>/Music/`, preserving the tree; missing or changed (size/mtime) files are copied, identical ones skipped. Same USB detection as `pos system backup` (lsblk TRAN + lsusb/by-id, mount offer for unmounted sticks, multi-stick picker). `--mp3`/`--mp4` filter by extension, neither = both; `--dry-run` previews. Config: `MEDIA_SYNC_SOURCE`, `MEDIA_SYNC_DEST`, shared `USB_MOUNT_BASE`/`USB_BYID` from `~/.config/linux_post_install/system.env`. Result notified via `lib/notify.sh`. Bare invocation on a terminal (or the `menu` subcommand) opens an interactive menu wrapping these actions (sync now mp3+mp4, dry-run preview, mp3-only, mp4-only, change source folder); flags stay scriptable | | `pos media ytsync [add\|sync\|list\|remove]` | `bin/pos-media-ytsync` | Incremental YouTube channel/playlist sync — first run asks for a URL (bare invocation = interactive menu; empty state goes straight to the prompt), repeat runs fetch only new videos | One yt-dlp call per new video (`bestvideo*+bestaudio/best` → MP4, metadata/chapters/thumbnail embedded, `--no-overwrites`, `--windows-filenames --trim-filenames 120`); per-source `--download-archive` (`~/.local/share/linux_post_install/ytsync/archive/<slug>.txt`) makes runs crash-safe and idempotent; registry tracks slug/type/url/subdir. Verbs never prompt (scheduler/timer safe); non-tty interactive entry prints a guard line and exits 0. `--dry-run` probes + plans with zero writes. Notify digest only when new>0 or failed>0 via `lib/notify.sh`. Config: `YTSYNC_VIDEOS_DIR`, `YTSYNC_EXTRA_ARGS` via `pos config ytsync`; automate with `pos system schedule` (`COMMAND=pos media ytsync sync`, `NOTIFY=never`) | diff --git a/DOC/howto/media.md b/DOC/howto/media.md index 5f75464..a63d099 100644 --- a/DOC/howto/media.md +++ b/DOC/howto/media.md @@ -3,26 +3,34 @@ Download audio and video from the web via `yt-dlp`, auto-classify URLs, sync your library to a USB stick, and keep YouTube channels incrementally up to date. -Tools: `grab`, `mp3`, `mp4`, `sync`, `ytsync`. +Tools: `yt mp3`, `yt mp4`, `yt grab`, `yt subtitles`, `sync`, `ytsync`. + +The YouTube download tools live under `pos media yt` (`mp3`, `mp4`, `grab`, +`subtitles`, `ytsync`). The legacy standalone names `pos media mp3`, `pos media +mp4` and `pos media grab` still work — they are thin forwarders to the `yt` +forms. | Tool | What it does | |------|--------------| -| `pos media grab` | Auto-classify URL and download as audio or video | -| `pos media mp3` | Download audio, convert to MP3 | -| `pos media mp4` | Download video with smart/interactive format selection | +| `pos media yt mp3` | Download audio, convert to MP3 | +| `pos media yt mp4` | Download video with smart/interactive format selection | +| `pos media yt grab` | Auto-classify URL and download as audio or video | +| `pos media yt subtitles` | Extract subtitles/captions from a URL | +| `pos media yt ytsync` | Forwarder → `pos media ytsync` | | `pos media sync` | Incrementally copy `~/Music` onto a USB stick (mp3/mp4) | | `pos media ytsync` | Track YouTube channels/playlists and download only new videos into `~/Videos` | -Requires `yt-dlp` and `ffmpeg` (`sudo apt install yt-dlp ffmpeg`); the tools -fail with a clean error message instead of a raw `command not found` if either -is missing. +Requires `yt-dlp` and `ffmpeg` (`sudo apt install yt-dlp ffmpeg`) for `mp3`/ +`mp4`/`grab`; `subtitles` needs only `yt-dlp` (no ffmpeg). The tools fail with a +clean error message instead of a raw `command not found` if a dependency is +missing. --- -## `pos media mp3` — audio as MP3 +## `pos media yt mp3` — audio as MP3 ```bash -pos media mp3 <url> +pos media yt mp3 <url> # or the legacy: pos media mp3 <url> ``` Extracts and converts the audio track to MP3 in `~/Music/`. With `--by-artist` @@ -30,8 +38,8 @@ files land in `~/Music/<artist>/<title>.mp3` (falls back to the uploader name when there's no artist tag), so a library stays organized. ```bash -pos media mp3 https://youtube.com/watch?v=dQw4w9WgXcQ -pos media mp3 --by-artist https://youtu.be/dQw4w9WgXcQ +pos media yt mp3 https://youtube.com/watch?v=dQw4w9WgXcQ +pos media yt mp3 --by-artist https://youtu.be/dQw4w9WgXcQ ``` MP3s are encoded at best quality with title/artist/album/date/chapters embedded @@ -48,15 +56,15 @@ cover art as a JPEG thumbnail. Existing files are never overwritten. **Recipe:** batch — loop over a list of URLs: ```bash -while read -r url; do pos media mp3 --by-artist "$url"; done < urls.txt +while read -r url; do pos media yt mp3 --by-artist "$url"; done < urls.txt ``` --- -## `pos media mp4` — video with smart format choice +## `pos media yt mp4` — video with smart format choice ```bash -pos media mp4 <url> +pos media yt mp4 <url> # or the legacy: pos media mp4 <url> ``` Without a format flag, the available formats are shown as a **short curated @@ -77,8 +85,8 @@ Non-interactive (scripting-friendly): | `--dry-run` | Print the yt-dlp command without running it | ```bash -pos media mp4 --best https://youtube.com/watch?v=dQw4w9WgXcQ -pos media mp4 -f 22 https://youtube.com/watch?v=dQw4w9WgXcQ +pos media yt mp4 --best https://youtube.com/watch?v=dQw4w9WgXcQ +pos media yt mp4 -f 22 https://youtube.com/watch?v=dQw4w9WgXcQ ``` Videos merge to MP4 with metadata, chapters, subtitles (all languages) and the @@ -90,13 +98,13 @@ video+audio and merges them. --- -## `pos media grab` — auto-classify URL and download +## `pos media yt grab` — auto-classify URL and download ```bash -pos media grab <url> +pos media yt grab <url> # or the legacy: pos media grab <url> ``` -Smart URL classifier that routes to `pos media mp3` or `pos media mp4` +Smart URL classifier that routes to `pos media yt mp3` or `pos media yt mp4` automatically based on the domain. Send a URL from your phone via Telegram and the bot downloads it to the right place without you thinking about it. @@ -116,16 +124,16 @@ unrecognized domains is `video` — change it with `pos config grab` or set `GRAB_DEFAULT=audio` in `~/.config/linux_post_install/grab.env`. ```bash -pos media grab https://music.youtube.com/watch?v=abc # → ~/Music -pos media grab https://youtube.com/watch?v=xyz # → ~/Videos -pos media grab --audio https://vimeo.com/123 # force mp3 -pos media grab --worst https://youtu.be/abc # lowest quality -pos media grab --dry-run https://soundcloud.com/artist/track # preview only +pos media yt grab https://music.youtube.com/watch?v=abc # → ~/Music +pos media yt grab https://youtube.com/watch?v=xyz # → ~/Videos +pos media yt grab --audio https://vimeo.com/123 # force mp3 +pos media yt grab --worst https://youtu.be/abc # lowest quality +pos media yt grab --dry-run https://soundcloud.com/artist/track # preview only ``` -Non-interactive by design — `pos media mp4` receives `--best` by default so it -never prompts for a format (critical for Telegram bot context where there's no -TTY). Pass `--worst` if you want the smallest file. +Non-interactive by design — `pos media yt mp4` receives `--best` by default so +it never prompts for a format (critical for Telegram bot context where there's +no TTY). Pass `--worst` if you want the smallest file. | Flag | Meaning | |------|---------| @@ -140,6 +148,39 @@ TTY). Pass `--worst` if you want the smallest file. --- +## `pos media yt subtitles` — extract subtitles/captions + +```bash +pos media yt subtitles <url> +``` + +Downloads subtitles/captions from a URL via yt-dlp. Fetches manual captions and +auto-generated captions by default (`--write-subs --write-auto-subs +--sub-langs best` — "best" picks the manually-created track when available, +otherwise the auto one). Output files land in the current directory as +`<title>.<lang>.<ext>`. + +```bash +pos media yt subtitles https://youtube.com/watch?v=dQw4w9WgXcQ +pos media yt subtitles --lang en https://youtu.be/dQw4w9WgXcQ +pos media yt subtitles --lang en,ar --format txt https://youtube.com/watch?v=dQw4w9WgXcQ +``` + +| Flag | Meaning | +|------|---------| +| `--lang <list>` | Subtitle languages, comma-separated (default `best`) — `en,ar` is passed as ONE `--sub-langs` arg | +| `--format <fmt>` | `srt` (default) / `vtt` / `txt` — `txt` converts srt→txt (timestamps, seq numbers and HTML tags stripped) | +| `--auto-only` | Only auto-generated captions (no manual subs) | +| `-o, --output <dir>` | Output directory (default: current directory) | +| `--list-subs` | List available subtitles for the URL and exit (probe only, no download) | +| `--no-playlist` | Download only the single video | +| `--dry-run` | Print the yt-dlp command without running it | + +If a video has no available subtitles the tool reports +`unavailable subtitles for this video (try --list-subs to check)`. + +--- + ## `pos media sync` — music onto a USB stick ```bash diff --git a/bin/pos b/bin/pos index 19e0b9c..cdaf007 100755 --- a/bin/pos +++ b/bin/pos @@ -266,7 +266,7 @@ MAIN_LOG="$LOG_DIR/pos.log" log_cmd() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $* → exit $2" >> "$MAIN_LOG"; } # Commands that read from stdin interactively — only log invocation -INTERACTIVE_CMDS="docker-compose docker-vbox network-hotspot system-firewall media-mp4 media-sync system-backup system-uninstall share-usb-server share-smb-server share-smb-client share-nfs-client share-nfs-server communication-telegram-listener communication-matrix-listener ai ai-gemini ai-openrouter ai-llamacpp ai-alias system-schedule entertainment-config config" +INTERACTIVE_CMDS="docker-compose docker-vbox network-hotspot system-firewall media-mp4 media-yt-mp4 media-sync system-backup system-uninstall share-usb-server share-smb-server share-smb-client share-nfs-client share-nfs-server communication-telegram-listener communication-matrix-listener ai ai-gemini ai-openrouter ai-llamacpp ai-alias system-schedule entertainment-config config" for ((i=n-1; i>=0; i--)); do cmd="pos" diff --git a/bin/pos-media-grab b/bin/pos-media-grab index d0da341..522c3a1 100755 --- a/bin/pos-media-grab +++ b/bin/pos-media-grab @@ -1,219 +1,5 @@ #!/usr/bin/env bash set -euo pipefail # POS: media grab — Auto-download URL as audio or video (classify + route) -# POS_FLAGS: --audio --video --best --worst --output --no-playlist --cookies --dry-run -# POS_CONFIG: grab | grab.env | GRAB_DEFAULT=:Default mode for unknown domains (video or audio, default video) - -source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" - -# Shared config loader (canonical KEY=VALUE parser, env-wins precedence) -source "$(dirname "$0")/../lib/config-ui.sh" 2>/dev/null || source "$(dirname "$0")/config-ui.sh" - -# Load grab.env config (env-seam: GRAB_DEFAULT) -load_grab_config() { - load_env_file "$CONFIG_DIR/grab.env" -} - -load_grab_config - -# ── URL classification ───────────────────────────────────────── -classify_url() { - local url="$1" mode="${GRAB_DEFAULT:-video}" - case "$url" in - *music.youtube.com*) echo "audio" ;; - *soundcloud.com*) echo "audio" ;; - *bandcamp.com*) echo "audio" ;; - *youtube.com*|*youtu.be*) echo "video" ;; - *vimeo.com*) echo "video" ;; - *twitch.tv*) echo "video" ;; - *) echo "$mode" ;; - esac -} - -usage() { - cat <<EOF -Usage: pos media grab [options] <url> - -Auto-download a URL as audio or video. Classifies the domain and delegates -to 'pos media mp3' (audio) or 'pos media mp4' (video). - -Options: - --audio Force audio (mp3) download - --video Force video (mp4) download - --best Best quality for video (default for non-interactive) - --worst Lowest quality for video - -o, --output <dir> Output directory (passed to mp3/mp4) - --no-playlist Download only the single video - --cookies <file> Netscape cookies.txt for age-gated content - --dry-run Print the command that would run, don't execute - -h, --help This help - -Examples: - pos media grab https://music.youtube.com/watch?v=abc - pos media grab https://youtube.com/watch?v=xyz - pos media grab --audio https://vimeo.com/123 - pos media grab --worst https://youtu.be/abc - pos media grab --dry-run https://soundcloud.com/artist/track -EOF - exit 0 -} - -# ── Arg parsing ──────────────────────────────────────────────── -URL="" -FORCE_AUDIO=0 -FORCE_VIDEO=0 -BEST=0 -WORST=0 -DRY_RUN=0 -EXTRA_ARGS=() - -while [[ $# -gt 0 ]]; do - case "$1" in - -h|--help) usage ;; - --audio) FORCE_AUDIO=1; shift ;; - --video) FORCE_VIDEO=1; shift ;; - --best) BEST=1; shift ;; - --worst) WORST=1; shift ;; - -o|--output) - [ $# -ge 2 ] || err "pos media grab: --output needs a value" - EXTRA_ARGS+=(--output "$2"); shift 2 ;; - --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; - --cookies) - [ $# -ge 2 ] || err "pos media grab: --cookies needs a value" - EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; - --dry-run) DRY_RUN=1; shift ;; - -*) err "pos media grab: Unknown option: $1 (see --help)" ;; - *) - [ -z "$URL" ] && URL="$1" && shift || err "pos media grab: Unexpected argument: $1" ;; - esac -done - -[ -n "$URL" ] || usage - -# Validate URL scheme -case "$URL" in - http://*|https://*) ;; - *) err "pos media grab: not a valid URL: $URL (must start with http:// or https://)" ;; -esac - -# Validate mutually exclusive overrides -[ "$FORCE_AUDIO" -eq 1 ] && [ "$FORCE_VIDEO" -eq 1 ] && \ - err "pos media grab: --audio and --video are mutually exclusive" -[ "$BEST" -eq 1 ] && [ "$WORST" -eq 1 ] && \ - err "pos media grab: --best and --worst are mutually exclusive" - -# ── Classification ───────────────────────────────────────────── -mode="" -if [ "$FORCE_AUDIO" -eq 1 ]; then - mode="audio" -elif [ "$FORCE_VIDEO" -eq 1 ]; then - mode="video" -else - mode="$(classify_url "$URL")" -fi - -# ── Build delegated command ──────────────────────────────────── -DELEGATE_ARGS=() - -if [ "$mode" = "audio" ]; then - DELEGATE_ARGS=(pos media mp3 "${EXTRA_ARGS[@]}") -else - # mp4 route: --best by default (non-interactive), --worst if user passes it - if [ "$WORST" -eq 1 ]; then - DELEGATE_ARGS=(pos media mp4 --worst "${EXTRA_ARGS[@]}") - else - DELEGATE_ARGS=(pos media mp4 --best "${EXTRA_ARGS[@]}") - fi -fi - -# ── Dry run ──────────────────────────────────────────────────── -if [ "$DRY_RUN" -eq 1 ]; then - echo "${DELEGATE_ARGS[*]} $URL" - exit 0 -fi - -# ── Execute ──────────────────────────────────────────────────── -output="" -rc=0 -if output=$("${DELEGATE_ARGS[@]}" "$URL" 2>&1); then - rc=0 -else - rc=$? -fi - -if [ "$rc" -ne 0 ]; then - # Summarize stderr for the user - summary="$(printf '%s' "$output" | grep -i 'error\|fail' | head -1 || true)" - [ -z "$summary" ] && summary="exit code $rc" - err "pos media grab: ❌ Download failed: $summary" -fi - -# ── Metadata + summary ──────────────────────────────────────── -title="" -duration="" -file_path="" - -# Determine expected output directory -if [ "$mode" = "audio" ]; then - out_dir="$HOME/Music" - for (( i=0; i<${#EXTRA_ARGS[@]}; i++ )); do - if [ "${EXTRA_ARGS[$i]}" = "--output" ] && [ $(( i + 1 )) -lt ${#EXTRA_ARGS[@]} ]; then - out_dir="${EXTRA_ARGS[$(( i + 1 ))]}" - break - fi - done - file_ext="mp3" -else - out_dir="$HOME/Videos" - for (( i=0; i<${#EXTRA_ARGS[@]}; i++ )); do - if [ "${EXTRA_ARGS[$i]}" = "--output" ] && [ $(( i + 1 )) -lt ${#EXTRA_ARGS[@]} ]; then - out_dir="${EXTRA_ARGS[$(( i + 1 ))]}" - break - fi - done - file_ext="mp4" -fi - -# Fetch metadata (fast, no download) -if command -v yt-dlp &>/dev/null; then - meta="$(yt-dlp --print title --print duration_string --no-warnings "$URL" 2>/dev/null || true)" - title="$(printf '%s' "$meta" | sed -n '1p')" - duration="$(printf '%s' "$meta" | sed -n '2p')" -fi - -# Find the downloaded file (most recent matching extension in out_dir) -if [ -d "$out_dir" ]; then - file_path="$(find "$out_dir" -maxdepth 1 -name "*.$file_ext" -printf '%T@ %p\n' 2>/dev/null \ - | sort -rn | head -1 | cut -d' ' -f2- || true)" -fi - -# Build summary -[ -z "$title" ] && title="$(basename "$URL" | sed 's/[?#].*//')" -[ -z "$duration" ] && duration="?" - -if [ "$mode" = "audio" ]; then - emoji="🎵" -else - emoji="🎬" -fi - -echo "$emoji Downloaded: $title ($duration)" - -if [ -n "$file_path" ] && [ -f "$file_path" ]; then - file_size="$(stat --printf='%s' "$file_path" 2>/dev/null || echo "0")" - # Format size in human-readable form - if [ "$file_size" -ge 1073741824 ]; then - size_human="$(awk "BEGIN { printf \"%.1f GB\", $file_size / 1073741824 }")" - elif [ "$file_size" -ge 1048576 ]; then - size_human="$(awk "BEGIN { printf \"%.1f MB\", $file_size / 1048576 }")" - elif [ "$file_size" -ge 1024 ]; then - size_human="$(awk "BEGIN { printf \"%.1f KB\", $file_size / 1024 }")" - else - size_human="${file_size} B" - fi - # Show path relative to HOME - rel_path="${file_path/#$HOME/\~}" - echo "📁 $rel_path ($size_human)" -else - echo "📁 $out_dir/ ($file_ext)" -fi +# -h|--help forwarded via exec below +exec pos media yt grab "$@" diff --git a/bin/pos-media-mp3 b/bin/pos-media-mp3 index 7e8830e..851578d 100755 --- a/bin/pos-media-mp3 +++ b/bin/pos-media-mp3 @@ -1,86 +1,5 @@ #!/usr/bin/env bash set -euo pipefail # POS: media mp3 — Download audio as MP3 (yt-dlp) -# POS_FLAGS: --output --no-playlist --cookies --by-artist --dry-run - -source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" - -# --dry-run is parsed before the deps guards (below): it previews the yt-dlp -# command without requiring yt-dlp/ffmpeg to be installed. -DRY_RUN=0 -for arg in "$@"; do - [ "$arg" = "--dry-run" ] && DRY_RUN=1 -done - -# Deps guards sit before -h|--help (help also errors on a box missing the deps). -if [ "$DRY_RUN" -eq 0 ]; then - command -v yt-dlp &>/dev/null || err "yt-dlp not found — install it with: sudo apt install yt-dlp" - command -v ffmpeg &>/dev/null || err "ffmpeg not found (needed for MP3 conversion) — install it with: sudo apt install ffmpeg" -fi - -OUT_DIR="$HOME/Music" -BY_ARTIST=0 -EXTRA_ARGS=() - -usage() { - cat <<EOF -Usage: pos media mp3 [options] <url> - -Download audio from a URL and convert to MP3 via yt-dlp. - -Options: - -o, --output <dir> Output directory (default: $OUT_DIR) - --by-artist Organize as <dir>/<artist>/<title>.mp3 - --no-playlist Download only the single video - --cookies <file> Netscape cookies.txt for age-gated content - --dry-run Print the yt-dlp command without running it - -h, --help This help - -Examples: - pos media mp3 https://youtube.com/watch?v=dQw4w9WgXcQ - pos media mp3 --by-artist --no-playlist https://youtu.be/dQw4w9WgXcQ - pos media mp3 --cookies ~/cookies.txt https://vimeo.com/123 -EOF - exit 0 -} - -URL="" -while [[ $# -gt 0 ]]; do - case "$1" in - -h|--help) usage ;; - -o|--output) - [ $# -ge 2 ] || err "--output needs a value" - OUT_DIR="$2"; shift 2 ;; - --by-artist) BY_ARTIST=1; shift ;; - --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; - --cookies) - [ $# -ge 2 ] || err "--cookies needs a value" - [ -f "$2" ] || err "cookies file not found: $2" - EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; - --dry-run) DRY_RUN=1; shift ;; - -*) err "Unknown option: $1 (see --help)" ;; - *) - [ -z "$URL" ] && URL="$1" && shift || err "Unexpected argument: $1" ;; - esac -done -[ -n "$URL" ] || usage - -if [ "$BY_ARTIST" -eq 1 ]; then - OUT_TEMPLATE="$OUT_DIR/%(artist,uploader)s/%(title)s.%(ext)s" -else - OUT_TEMPLATE="$OUT_DIR/%(title)s.%(ext)s" -fi - -args=(yt-dlp -x --audio-format mp3 --audio-quality 0 - --embed-metadata --embed-chapters - --embed-thumbnail --convert-thumbnails jpg - --parse-metadata "%(artist,uploader)s:%(artist)s" - --no-overwrites - -o "$OUT_TEMPLATE" - "${EXTRA_ARGS[@]}" "$URL") - -if [ "$DRY_RUN" -eq 1 ]; then - echo "yt-dlp ${args[*]:1}" -else - spawn "downloading audio → $OUT_DIR" "${args[@]}" -fi +# -h|--help forwarded via exec below +exec pos media yt mp3 "$@" diff --git a/bin/pos-media-mp4 b/bin/pos-media-mp4 index 92be17f..d9f04a7 100755 --- a/bin/pos-media-mp4 +++ b/bin/pos-media-mp4 @@ -1,132 +1,5 @@ #!/usr/bin/env bash set -euo pipefail # POS: media mp4 — Download video as MP4 (smart/interactive format select) -# POS_FLAGS: --format --best --worst --output --no-playlist --cookies --dry-run - -source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" - -# --dry-run is parsed before the deps guards (below): it previews the yt-dlp -# command without requiring yt-dlp/ffmpeg to be installed. -DRY_RUN=0 -for arg in "$@"; do - [ "$arg" = "--dry-run" ] && DRY_RUN=1 -done - -# Deps guards sit before -h|--help (help also errors on a box missing the deps). -if [ "$DRY_RUN" -eq 0 ]; then - command -v yt-dlp &>/dev/null || err "yt-dlp not found — install it with: sudo apt install yt-dlp" - command -v ffmpeg &>/dev/null || err "ffmpeg not found (needed for MP4 merge) — install it with: sudo apt install ffmpeg" -fi - -OUT_DIR="$HOME/Videos" -FORMAT="" -BEST=0 -WORST=0 -EXTRA_ARGS=() - -usage() { - cat <<EOF -Usage: pos media mp4 [options] <url> - -Download video from a URL via yt-dlp. Without -f/--best/--worst, formats are -listed and picked interactively. - -Options: - -f, --format <id> Download that format id directly (no prompt) - --best Best video + audio (no prompt) - --worst Lowest quality (no prompt) - -o, --output <dir> Output directory (default: $OUT_DIR) - --no-playlist Download only the single video - --cookies <file> Netscape cookies.txt for age-gated content - --dry-run Print the yt-dlp command without running it - -h, --help This help - -Examples: - pos media mp4 https://youtube.com/watch?v=dQw4w9WgXcQ - pos media mp4 --best https://youtube.com/watch?v=dQw4w9WgXcQ - pos media mp4 -f 22 https://youtube.com/watch?v=dQw4w9WgXcQ -EOF - exit 0 -} - -URL="" -while [[ $# -gt 0 ]]; do - case "$1" in - -h|--help) usage ;; - -f|--format) - [ $# -ge 2 ] || err "--format needs a value" - FORMAT="$2"; shift 2 ;; - --best) BEST=1; shift ;; - --worst) WORST=1; shift ;; - -o|--output) - [ $# -ge 2 ] || err "--output needs a value" - OUT_DIR="$2"; shift 2 ;; - --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; - --cookies) - [ $# -ge 2 ] || err "--cookies needs a value" - [ -f "$2" ] || err "cookies file not found: $2" - EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; - --dry-run) DRY_RUN=1; shift ;; - -*) err "Unknown option: $1 (see --help)" ;; - *) - [ -z "$URL" ] && URL="$1" && shift || err "Unexpected argument: $1" ;; - esac -done -[ -n "$URL" ] || usage - -[ -n "$FORMAT" ] && { [ "$BEST" -eq 1 ] || [ "$WORST" -eq 1 ]; } \ - && err "--format and --best/--worst are mutually exclusive" -[ "$BEST" -eq 1 ] && [ "$WORST" -eq 1 ] && err "--best and --worst are mutually exclusive" - -# Resolve the format selector: explicit flag → smart preset → interactive. -# The curated table goes to stderr; stdout carries ONLY the chosen format id -# (same ui_pick lesson — a $(...) capture must not swallow the menu). -fmt_spec="" -if [ -n "$FORMAT" ]; then - fmt_spec="$FORMAT" -elif [ "$BEST" -eq 1 ]; then - fmt_spec="bestvideo*+bestaudio/best" -elif [ "$WORST" -eq 1 ]; then - fmt_spec="worst" -else - table="$(yt-dlp -F "$URL" 2>&1)" || { - echo "ERROR: no formats listed (unsupported site or age-gated content)" >&2 - exit 1 - } - printf '%s' "$table" | awk -F'[[:space:]][[:space:]]+' ' - NR <= 2 { print; next } - $0 ~ /^-+$/ { print; next } - $0 ~ /audio only/ && $0 !~ /storyboard/ && $1 !~ /^sb/ { print " [audio] " $0; next } - $0 ~ /video only/ && $1 !~ /^sb/ { print " [video] " $0; next } - $1 ~ /^[0-9]+$/ && $2 ~ /^(mp4|webm)$/ && $3 ~ /x/ { print " [combo] " $0 } - ' >&2 - echo >&2 - read -rp "Format ID ('best'/'worst'/empty = best): " chosen - case "${chosen:-best}" in - best) fmt_spec="bestvideo*+bestaudio/best" ;; - worst) fmt_spec="worst" ;; - *) - if printf '%s' "$table" | grep -qE "^[[:space:]]*${chosen}[[:space:]]"; then - fmt_spec="$chosen" - else - echo "ERROR: unknown format id '$chosen'" >&2 - exit 1 - fi - ;; - esac -fi - -args=(yt-dlp -f "$fmt_spec" - --merge-output-format mp4 - --embed-metadata --embed-chapters - --embed-subs --sub-langs all - --embed-thumbnail - --no-overwrites - -o "$OUT_DIR/%(title)s.%(ext)s" - "${EXTRA_ARGS[@]}" "$URL") - -if [ "$DRY_RUN" -eq 1 ]; then - echo "yt-dlp ${args[*]:1}" -else - spawn "downloading video → $OUT_DIR" "${args[@]}" -fi +# -h|--help forwarded via exec below +exec pos media yt mp4 "$@" diff --git a/bin/pos-media-yt b/bin/pos-media-yt new file mode 100755 index 0000000..38ff474 --- /dev/null +++ b/bin/pos-media-yt @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt — YouTube download tools (mp3/mp4/grab/ytsync/subtitles) +# POS_SUBCMDS: mp3 mp4 grab ytsync subtitles +source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" + +usage() { + cat <<'EOF' +Usage: pos media yt <command> [args] + +YouTube download tools: audio, video, auto-grab, subtitles, channel sync. + +Commands: + mp3 <url> Download audio as MP3 + mp4 <url> Download video as MP4 (interactive format select) + grab <url> Auto-download as audio or video + ytsync [cmd] Incremental YouTube channel sync + subtitles <url> Extract subtitles/captions + +Options: + -h, --help This help + +Run 'pos media yt <command> --help' for details. +EOF + exit 0 +} + +case "${1:-}" in + -h|--help|"") usage ;; +esac +# Subcommands are resolved by the dispatcher via pos-media-yt-<sub> files. +# This file only handles bare 'pos media yt' (help) and unknown args. +err "unknown yt command: $1 (see 'pos media yt --help')" diff --git a/bin/pos-media-yt-grab b/bin/pos-media-yt-grab new file mode 100755 index 0000000..19cf28d --- /dev/null +++ b/bin/pos-media-yt-grab @@ -0,0 +1,208 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt-grab — Auto-download URL as audio or video (classify + route) +# POS_FLAGS: --audio --video --best --worst --output --no-playlist --cookies --dry-run +# POS_CONFIG: grab | grab.env | GRAB_DEFAULT=:Default mode for unknown domains (video or audio, default video) + +source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" + +# Shared config loader (canonical KEY=VALUE parser, env-wins precedence) +source "$(dirname "$0")/../lib/config-ui.sh" 2>/dev/null || source "$(dirname "$0")/config-ui.sh" + +# Shared yt helpers (classify_url, yt_validate_url, yt_check_deps, yt_echo_cmd) +source "$(dirname "$0")/../lib/yt-lib.sh" 2>/dev/null || source "$(dirname "$0")/yt-lib.sh" + +# Load grab.env config (env-seam: GRAB_DEFAULT) +load_grab_config() { + load_env_file "$CONFIG_DIR/grab.env" +} + +load_grab_config + +usage() { + cat <<EOF +Usage: pos media yt grab [options] <url> + +Auto-download a URL as audio or video. Classifies the domain and delegates +to 'pos media yt mp3' (audio) or 'pos media yt mp4' (video). + +Options: + --audio Force audio (mp3) download + --video Force video (mp4) download + --best Best quality for video (default for non-interactive) + --worst Lowest quality for video + -o, --output <dir> Output directory (passed to mp3/mp4) + --no-playlist Download only the single video + --cookies <file> Netscape cookies.txt for age-gated content + --dry-run Print the command that would run, don't execute + -h, --help This help + +Examples: + pos media yt grab https://music.youtube.com/watch?v=abc + pos media yt grab https://youtube.com/watch?v=xyz + pos media yt grab --audio https://vimeo.com/123 + pos media yt grab --worst https://youtu.be/abc + pos media yt grab --dry-run https://soundcloud.com/artist/track + +The legacy spelling 'pos media grab' still works (forwarder) and is equivalent. +EOF + exit 0 +} + +# ── Arg parsing ──────────────────────────────────────────────── +URL="" +FORCE_AUDIO=0 +FORCE_VIDEO=0 +BEST=0 +WORST=0 +DRY_RUN=0 +EXTRA_ARGS=() + +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage ;; + --audio) FORCE_AUDIO=1; shift ;; + --video) FORCE_VIDEO=1; shift ;; + --best) BEST=1; shift ;; + --worst) WORST=1; shift ;; + -o|--output) + [ $# -ge 2 ] || err "yt-grab: --output needs a value" + EXTRA_ARGS+=(--output "$2"); shift 2 ;; + --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; + --cookies) + [ $# -ge 2 ] || err "yt-grab: --cookies needs a value" + EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -*) err "yt-grab: Unknown option: $1 (see --help)" ;; + *) + [ -z "$URL" ] && URL="$1" && shift || err "yt-grab: Unexpected argument: $1" ;; + esac +done + +[ -n "$URL" ] || usage + +# Validate URL scheme +yt_validate_url "$URL" 2>/dev/null || \ + err "yt-grab: not a valid URL: $URL (must start with http:// or https://)" + +# Validate mutually exclusive overrides +[ "$FORCE_AUDIO" -eq 1 ] && [ "$FORCE_VIDEO" -eq 1 ] && \ + err "yt-grab: --audio and --video are mutually exclusive" +[ "$BEST" -eq 1 ] && [ "$WORST" -eq 1 ] && \ + err "yt-grab: --best and --worst are mutually exclusive" + +# ── Classification ───────────────────────────────────────────── +mode="" +if [ "$FORCE_AUDIO" -eq 1 ]; then + mode="audio" +elif [ "$FORCE_VIDEO" -eq 1 ]; then + mode="video" +else + mode="$(classify_url "$URL")" +fi + +# ── Build delegated command ──────────────────────────────────── +DELEGATE_ARGS=() + +if [ "$mode" = "audio" ]; then + DELEGATE_ARGS=(pos media yt mp3 "${EXTRA_ARGS[@]}") +else + # mp4 route: --best by default (non-interactive), --worst if user passes it + if [ "$WORST" -eq 1 ]; then + DELEGATE_ARGS=(pos media yt mp4 --worst "${EXTRA_ARGS[@]}") + else + DELEGATE_ARGS=(pos media yt mp4 --best "${EXTRA_ARGS[@]}") + fi +fi + +# ── Dry run ──────────────────────────────────────────────────── +if [ "$DRY_RUN" -eq 1 ]; then + echo "${DELEGATE_ARGS[*]} $URL" + exit 0 +fi + +# ── Execute ──────────────────────────────────────────────────── +output="" +rc=0 +if output=$("${DELEGATE_ARGS[@]}" "$URL" 2>&1); then + rc=0 +else + rc=$? +fi + +if [ "$rc" -ne 0 ]; then + # Summarize stderr for the user + summary="$(printf '%s' "$output" | grep -i 'error\|fail' | head -1 || true)" + [ -z "$summary" ] && summary="exit code $rc" + err "yt-grab: ❌ Download failed: $summary" +fi + +# ── Metadata + summary ──────────────────────────────────────── +title="" +duration="" +file_path="" + +# Determine expected output directory +if [ "$mode" = "audio" ]; then + out_dir="$HOME/Music" + for (( i=0; i<${#EXTRA_ARGS[@]}; i++ )); do + if [ "${EXTRA_ARGS[$i]}" = "--output" ] && [ $(( i + 1 )) -lt ${#EXTRA_ARGS[@]} ]; then + out_dir="${EXTRA_ARGS[$(( i + 1 ))]}" + break + fi + done + file_ext="mp3" +else + out_dir="$HOME/Videos" + for (( i=0; i<${#EXTRA_ARGS[@]}; i++ )); do + if [ "${EXTRA_ARGS[$i]}" = "--output" ] && [ $(( i + 1 )) -lt ${#EXTRA_ARGS[@]} ]; then + out_dir="${EXTRA_ARGS[$(( i + 1 ))]}" + break + fi + done + file_ext="mp4" +fi + +# Fetch metadata (fast, no download) +if command -v yt-dlp &>/dev/null; then + meta="$(yt-dlp --print title --print duration_string --no-warnings "$URL" 2>/dev/null || true)" + title="$(printf '%s' "$meta" | sed -n '1p')" + duration="$(printf '%s' "$meta" | sed -n '2p')" +fi + +# Find the downloaded file (most recent matching extension in out_dir) +if [ -d "$out_dir" ]; then + file_path="$(find "$out_dir" -maxdepth 1 -name "*.$file_ext" -printf '%T@ %p\n' 2>/dev/null \ + | sort -rn | head -1 | cut -d' ' -f2- || true)" +fi + +# Build summary +[ -z "$title" ] && title="$(basename "$URL" | sed 's/[?#].*//')" +[ -z "$duration" ] && duration="?" + +if [ "$mode" = "audio" ]; then + emoji="🎵" +else + emoji="🎬" +fi + +echo "$emoji Downloaded: $title ($duration)" + +if [ -n "$file_path" ] && [ -f "$file_path" ]; then + file_size="$(stat --printf='%s' "$file_path" 2>/dev/null || echo "0")" + # Format size in human-readable form + if [ "$file_size" -ge 1073741824 ]; then + size_human="$(awk "BEGIN { printf \"%.1f GB\", $file_size / 1073741824 }")" + elif [ "$file_size" -ge 1048576 ]; then + size_human="$(awk "BEGIN { printf \"%.1f MB\", $file_size / 1048576 }")" + elif [ "$file_size" -ge 1024 ]; then + size_human="$(awk "BEGIN { printf \"%.1f KB\", $file_size / 1024 }")" + else + size_human="${file_size} B" + fi + # Show path relative to HOME + rel_path="${file_path/#$HOME/\~}" + echo "📁 $rel_path ($size_human)" +else + echo "📁 $out_dir/ ($file_ext)" +fi diff --git a/bin/pos-media-yt-mp3 b/bin/pos-media-yt-mp3 new file mode 100755 index 0000000..7231799 --- /dev/null +++ b/bin/pos-media-yt-mp3 @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt-mp3 — Download audio as MP3 (yt-dlp) +# POS_FLAGS: --output --no-playlist --cookies --by-artist --dry-run + +source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" +source "$(dirname "$0")/../lib/yt-lib.sh" 2>/dev/null || source "$(dirname "$0")/yt-lib.sh" + +# --dry-run is parsed before the deps guards (below): it previews the yt-dlp +# command without requiring yt-dlp/ffmpeg to be installed. +DRY_RUN=0 +for arg in "$@"; do + [ "$arg" = "--dry-run" ] && DRY_RUN=1 +done + +# Deps guards sit before -h|--help (help also errors on a box missing the deps). +# yt_check_deps with dry-run=1 skips the dep checks (dry-run preview needs none). +yt_check_deps "$DRY_RUN" "MP3 conversion" + +# Env seam: YT_OUT_DIR overrides the default (keeps HOME sandbox-compatible). +OUT_DIR="${YT_OUT_DIR:-$HOME/Music}" +BY_ARTIST=0 +EXTRA_ARGS=() + +usage() { + cat <<EOF +Usage: pos media yt mp3 [options] <url> + +Download audio from a URL and convert to MP3 via yt-dlp. + +Options: + -o, --output <dir> Output directory (default: $OUT_DIR) + --by-artist Organize as <dir>/<artist>/<title>.mp3 + --no-playlist Download only the single video + --cookies <file> Netscape cookies.txt for age-gated content + --dry-run Print the yt-dlp command without running it + -h, --help This help + +Examples: + pos media yt mp3 https://youtube.com/watch?v=dQw4w9WgXcQ + pos media yt mp3 --by-artist --no-playlist https://youtu.be/dQw4w9WgXcQ + pos media yt mp3 --cookies ~/cookies.txt https://vimeo.com/123 +EOF + exit 0 +} + +URL="" +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage ;; + -o|--output) + [ $# -ge 2 ] || err "--output needs a value" + OUT_DIR="$2"; shift 2 ;; + --by-artist) BY_ARTIST=1; shift ;; + --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; + --cookies) + [ $# -ge 2 ] || err "--cookies needs a value" + [ -f "$2" ] || err "cookies file not found: $2" + EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -*) err "Unknown option: $1 (see --help)" ;; + *) + [ -z "$URL" ] && URL="$1" && shift || err "Unexpected argument: $1" ;; + esac +done +[ -n "$URL" ] || usage + +if [ "$BY_ARTIST" -eq 1 ]; then + OUT_TEMPLATE="$OUT_DIR/%(artist,uploader)s/%(title)s.%(ext)s" +else + OUT_TEMPLATE="$OUT_DIR/%(title)s.%(ext)s" +fi + +args=(yt-dlp -x --audio-format mp3 --audio-quality 0 + --embed-metadata --embed-chapters + --embed-thumbnail --convert-thumbnails jpg + --parse-metadata "%(artist,uploader)s:%(artist)s" + --no-overwrites + -o "$OUT_TEMPLATE" + "${EXTRA_ARGS[@]}" "$URL") + +if [ "$DRY_RUN" -eq 1 ]; then + echo "yt-dlp ${args[*]:1}" +else + spawn "downloading audio → $OUT_DIR" "${args[@]}" +fi diff --git a/bin/pos-media-yt-mp4 b/bin/pos-media-yt-mp4 new file mode 100755 index 0000000..a18eb5d --- /dev/null +++ b/bin/pos-media-yt-mp4 @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt-mp4 — Download video as MP4 (smart/interactive format select) +# POS_FLAGS: --format --best --worst --output --no-playlist --cookies --dry-run + +source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" +source "$(dirname "$0")/../lib/yt-lib.sh" 2>/dev/null || source "$(dirname "$0")/yt-lib.sh" + +# --dry-run is parsed before the deps guards (below): it previews the yt-dlp +# command without requiring yt-dlp/ffmpeg to be installed. +DRY_RUN=0 +for arg in "$@"; do + [ "$arg" = "--dry-run" ] && DRY_RUN=1 +done + +# Deps guards sit before -h|--help (help also errors on a box missing the deps). +# yt_check_deps with dry-run=1 skips the dep checks (dry-run preview needs none). +yt_check_deps "$DRY_RUN" "MP4 merge" + +# Env seam: YT_OUT_DIR overrides the default (keeps HOME sandbox-compatible). +OUT_DIR="${YT_OUT_DIR:-$HOME/Videos}" +FORMAT="" +BEST=0 +WORST=0 +EXTRA_ARGS=() + +usage() { + cat <<EOF +Usage: pos media yt mp4 [options] <url> + +Download video from a URL via yt-dlp. Without -f/--best/--worst, formats are +listed and picked interactively. + +Options: + -f, --format <id> Download that format id directly (no prompt) + --best Best video + audio (no prompt) + --worst Lowest quality (no prompt) + -o, --output <dir> Output directory (default: $OUT_DIR) + --no-playlist Download only the single video + --cookies <file> Netscape cookies.txt for age-gated content + --dry-run Print the yt-dlp command without running it + -h, --help This help + +Examples: + pos media yt mp4 https://youtube.com/watch?v=dQw4w9WgXcQ + pos media yt mp4 --best https://youtube.com/watch?v=dQw4w9WgXcQ + pos media yt mp4 -f 22 https://youtube.com/watch?v=dQw4w9WgXcQ +EOF + exit 0 +} + +URL="" +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage ;; + -f|--format) + [ $# -ge 2 ] || err "--format needs a value" + FORMAT="$2"; shift 2 ;; + --best) BEST=1; shift ;; + --worst) WORST=1; shift ;; + -o|--output) + [ $# -ge 2 ] || err "--output needs a value" + OUT_DIR="$2"; shift 2 ;; + --no-playlist) EXTRA_ARGS+=(--no-playlist); shift ;; + --cookies) + [ $# -ge 2 ] || err "--cookies needs a value" + [ -f "$2" ] || err "cookies file not found: $2" + EXTRA_ARGS+=(--cookies "$2"); shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -*) err "Unknown option: $1 (see --help)" ;; + *) + [ -z "$URL" ] && URL="$1" && shift || err "Unexpected argument: $1" ;; + esac +done +[ -n "$URL" ] || usage + +[ -n "$FORMAT" ] && { [ "$BEST" -eq 1 ] || [ "$WORST" -eq 1 ]; } \ + && err "--format and --best/--worst are mutually exclusive" +[ "$BEST" -eq 1 ] && [ "$WORST" -eq 1 ] && err "--best and --worst are mutually exclusive" + +# Resolve the format selector: explicit flag → smart preset → interactive. +# The curated table goes to stderr; stdout carries ONLY the chosen format id +# (same ui_pick lesson — a $(...) capture must not swallow the menu). +fmt_spec="" +if [ -n "$FORMAT" ]; then + fmt_spec="$FORMAT" +elif [ "$BEST" -eq 1 ]; then + fmt_spec="bestvideo*+bestaudio/best" +elif [ "$WORST" -eq 1 ]; then + fmt_spec="worst" +else + table="$(yt-dlp -F "$URL" 2>&1)" || { + echo "ERROR: no formats listed (unsupported site or age-gated content)" >&2 + exit 1 + } + printf '%s' "$table" | awk -F'[[:space:]][[:space:]]+' ' + NR <= 2 { print; next } + $0 ~ /^-+$/ { print; next } + $0 ~ /audio only/ && $0 !~ /storyboard/ && $1 !~ /^sb/ { print " [audio] " $0; next } + $0 ~ /video only/ && $1 !~ /^sb/ { print " [video] " $0; next } + $1 ~ /^[0-9]+$/ && $2 ~ /^(mp4|webm)$/ && $3 ~ /x/ { print " [combo] " $0 } + ' >&2 + echo >&2 + read -rp "Format ID ('best'/'worst'/empty = best): " chosen + case "${chosen:-best}" in + best) fmt_spec="bestvideo*+bestaudio/best" ;; + worst) fmt_spec="worst" ;; + *) + if printf '%s' "$table" | grep -qE "^[[:space:]]*${chosen}[[:space:]]"; then + fmt_spec="$chosen" + else + echo "ERROR: unknown format id '$chosen'" >&2 + exit 1 + fi + ;; + esac +fi + +args=(yt-dlp -f "$fmt_spec" + --merge-output-format mp4 + --embed-metadata --embed-chapters + --embed-subs --sub-langs all + --embed-thumbnail + --no-overwrites + -o "$OUT_DIR/%(title)s.%(ext)s" + "${EXTRA_ARGS[@]}" "$URL") + +if [ "$DRY_RUN" -eq 1 ]; then + echo "yt-dlp ${args[*]:1}" +else + spawn "downloading video → $OUT_DIR" "${args[@]}" +fi diff --git a/bin/pos-media-yt-subtitles b/bin/pos-media-yt-subtitles new file mode 100755 index 0000000..36dde4c --- /dev/null +++ b/bin/pos-media-yt-subtitles @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt-subtitles — Extract subtitles/captions from a URL (yt-dlp) +# POS_FLAGS: --lang --format --auto-only --output --list-subs --no-playlist --dry-run + +source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" +source "$(dirname "$0")/../lib/yt-lib.sh" 2>/dev/null || source "$(dirname "$0")/yt-lib.sh" + +# --dry-run / --list-subs are parsed before the deps guards (below): dry-run +# previews the yt-dlp command without requiring yt-dlp to be installed. +DRY_RUN=0 +LIST_SUBS=0 +for arg in "$@"; do + [ "$arg" = "--dry-run" ] && DRY_RUN=1 + [ "$arg" = "--list-subs" ] && LIST_SUBS=1 +done + +# Deps guards sit before -h|--help. Subtitles needs only yt-dlp (no ffmpeg). +# dry-run skips the deps check entirely (preview needs no binary). +if [ "$DRY_RUN" -eq 0 ]; then + command -v yt-dlp &>/dev/null || err "yt-dlp not found — install it with: sudo apt install yt-dlp" +fi + +LANG_SPEC="" +FMT="srt" +AUTO_ONLY=0 +OUT_DIR="" +NO_PLAYLIST=0 + +usage() { + cat <<EOF +Usage: pos media yt subtitles [options] <url> + +Extract subtitles/captions from a URL via yt-dlp. Fetches manual captions and +auto-generated captions by default. + +Options: + --lang <list> Subtitle languages (comma-separated), default: best + --format <fmt> Output format: srt (default), vtt, or txt + --auto-only Only auto-generated captions (no manual subs) + -o, --output <dir> Output directory (default: current directory) + --list-subs List available subtitles and exit (probe only) + --no-playlist Download only the single video + --dry-run Print the yt-dlp command without running it + -h, --help This help + +Examples: + pos media yt subtitles https://youtube.com/watch?v=dQw4w9WgXcQ + pos media yt subtitles --lang en,ar https://youtube.com/watch?v=dQw4w9WgXcQ + pos media yt subtitles --format txt https://youtu.be/dQw4w9WgXcQ + pos media yt subtitles --list-subs https://youtube.com/watch?v=dQw4w9WgXcQ +EOF + exit 0 +} + +# convert_srt_to_txt <srt-file> — strip SRT timestamps, seq numbers, HTML tags +convert_srt_to_txt() { + local srt_file="$1" txt_file="${1%.srt}.txt" + sed '/^[0-9][0-9]:[0-9][0-9]/d; /^$/d; s/<[^>]*>//g' "$srt_file" > "$txt_file" +} + +URL="" +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage ;; + --lang) + [ $# -ge 2 ] || err "--lang needs a value" + LANG_SPEC="$2"; shift 2 ;; + --format) + [ $# -ge 2 ] || err "--format needs a value" + case "$2" in + srt|vtt|txt) FMT="$2" ;; + *) err "unknown format '$2' (use srt, vtt, or txt)" ;; + esac + shift 2 ;; + --auto-only) AUTO_ONLY=1; shift ;; + -o|--output) + [ $# -ge 2 ] || err "--output needs a value" + OUT_DIR="$2"; shift 2 ;; + --list-subs) LIST_SUBS=1; shift ;; + --no-playlist) NO_PLAYLIST=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + -*) err "Unknown option: $1 (see --help)" ;; + *) + [ -z "$URL" ] && URL="$1" && shift || err "Unexpected argument: $1" ;; + esac +done +[ -n "$URL" ] || err "missing URL (see --help)" + +yt_validate_url "$URL" 2>/dev/null || err "not a valid URL: $URL (must start with http:// or https://)" + +# ── Build the yt-dlp command ───────────────────────────────── +if [ "$LIST_SUBS" -eq 1 ]; then + args=(yt-dlp --list-subs "$URL") + if [ "$DRY_RUN" -eq 1 ]; then + yt_echo_cmd "${args[@]:1}" + exit 0 + fi + spawn "listing subtitles" "${args[@]}" + exit 0 +fi + +# Output template +OUT_TEMPLATE="" +if [ -n "$OUT_DIR" ]; then + OUT_TEMPLATE="$OUT_DIR/%(title)s.%(sub_lang)s.%(ext)s" +else + OUT_TEMPLATE="./%(title)s.%(sub_lang)s.%(ext)s" +fi + +args=(yt-dlp) +if [ "$AUTO_ONLY" -eq 1 ]; then + args+=(--write-auto-subs) +else + args+=(--write-subs --write-auto-subs) +fi + +if [ -n "$LANG_SPEC" ]; then + args+=(--sub-langs "$LANG_SPEC") +else + args+=(--sub-langs best) +fi + +case "$FMT" in + vtt) args+=(--sub-format vtt) ;; + txt) args+=(--sub-format srt) ;; + *) args+=(--sub-format srt) ;; +esac + +[ "$NO_PLAYLIST" -eq 1 ] && args+=(--no-playlist) + +args+=(-o "$OUT_TEMPLATE" "$URL") + +if [ "$DRY_RUN" -eq 1 ]; then + yt_echo_cmd "${args[@]:1}" + exit 0 +fi + +# ── Execute ─────────────────────────────────────────────────── +set +e +out="$(mktemp)" +err_tmp="$(mktemp)" +"${args[@]}" >"$out" 2>"$err_tmp" +rc=$? +set -e + +if [ "$rc" -ne 0 ]; then + # Unavailable-subtitle detection: exit!=0 AND stderr matches no-subs patterns + if grep -qiE 'no subtitles|no captions|unable to extract|subtitles not available' "$err_tmp"; then + rm -f "$out" "$err_tmp" + err "unavailable subtitles for this video (try --list-subs to check)" + fi + # Re-surface the real yt-dlp error otherwise + sed 's/^/ /' "$err_tmp" >&2 + rm -f "$out" "$err_tmp" + exit "$rc" +fi +rm -f "$out" "$err_tmp" + +# ── Post-conversion for txt format ──────────────────────────── +if [ "$FMT" = "txt" ]; then + search_dir="." + [ -n "$OUT_DIR" ] && search_dir="$OUT_DIR" + while IFS= read -r -d '' f; do + case "$f" in + *.srt) convert_srt_to_txt "$f" ;; + esac + done < <(find "$search_dir" -maxdepth 1 -name '*.srt' -print0 2>/dev/null || true) +fi diff --git a/bin/pos-media-yt-ytsync b/bin/pos-media-yt-ytsync new file mode 100755 index 0000000..be50af0 --- /dev/null +++ b/bin/pos-media-yt-ytsync @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail +# POS: media yt-ytsync — Incrementally sync YouTube channels/playlists into ~/Videos +# -h|--help forwarded via exec below +exec pos media ytsync "$@" diff --git a/completions/pos.bash b/completions/pos.bash index f738950..a305863 100644 --- a/completions/pos.bash +++ b/completions/pos.bash @@ -11,10 +11,11 @@ _pos_flags[communication-telegram-sender]="--type --caption --parse-mode --no-pr _pos_flags[docker-stack]="-a --all" _pos_flags[docker-vbox]="--dir --gpu --device --port --cpus --memory --network" _pos_flags[entertainment-send]="--print --markdown" -_pos_flags[media-grab]="--audio --video --best --worst --output --no-playlist --cookies --dry-run" -_pos_flags[media-mp3]="--output --no-playlist --cookies --by-artist --dry-run" -_pos_flags[media-mp4]="--format --best --worst --output --no-playlist --cookies --dry-run" _pos_flags[media-sync]="--mp3 --mp4 --source --dry-run" +_pos_flags[media-yt-grab]="--audio --video --best --worst --output --no-playlist --cookies --dry-run" +_pos_flags[media-yt-mp3]="--output --no-playlist --cookies --by-artist --dry-run" +_pos_flags[media-yt-mp4]="--format --best --worst --output --no-playlist --cookies --dry-run" +_pos_flags[media-yt-subtitles]="--lang --format --auto-only --output --list-subs --no-playlist --dry-run" _pos_flags[media-ytsync]="--dry-run" _pos_flags[network-checkport]="--tcp --udp --ping --no-banner --versions --timeout" _pos_flags[network-download]="--dir --out --split --seed --force --upload --gid --tmux" @@ -42,6 +43,7 @@ _pos_subcmds[docker-compose]="ls installed up down restart logs update config me _pos_subcmds[docker-vbox]="create enter stop start rm ls menu" _pos_subcmds[media-sync]="menu" _pos_subcmds[media-ytsync]="add sync list remove" +_pos_subcmds[media-yt]="mp3 mp4 grab ytsync subtitles" _pos_subcmds[network-download]="start stop status add torrent metalink list info files peers pause resume remove purge move limit set watch restart retry replace menu" _pos_subcmds[share-nfs-client]="mount unmount list persist unpersist menu" _pos_subcmds[share-nfs-server]="status share unshare list reload enable disable menu" diff --git a/lib/yt-lib.sh b/lib/yt-lib.sh new file mode 100644 index 0000000..e7246bd --- /dev/null +++ b/lib/yt-lib.sh @@ -0,0 +1,50 @@ +# lib/yt-lib.sh — shared helpers for yt-dlp-based media tools. +# Sourced opt-in by yt-* subcommand files. Uses err() from lib/common.sh. + +# yt_check_deps [dry_run] [ffmpeg_purpose] +# Check yt-dlp (+ ffmpeg unless dry_run=1 and caller wants dry-run to skip deps). +# ffmpeg_purpose is the parenthetical in the error message (default: "MP3 conversion"). +# ytsync overrides this deliberately (needs yt-dlp+jq even in dry-run). +yt_check_deps() { + local dry="${1:-0}" purpose="${2:-MP3 conversion}" + command -v yt-dlp &>/dev/null || err "yt-dlp not found — install it with: sudo apt install yt-dlp" + if [ "$dry" -eq 0 ]; then + command -v ffmpeg &>/dev/null || err "ffmpeg not found (needed for $purpose) — install it with: sudo apt install ffmpeg" + fi +} + +# yt_validate_url <url> +# Returns 1 (does NOT exit) when URL doesn't start with http:// or https://; +# prints the diagnostic to stderr. Callers surface the error themselves (e.g. +# `yt_validate_url "$url" 2>/dev/null || err "…"` with their own prefix) so a +# helper whose only job is to check can never kill the caller's error path. +yt_validate_url() { + local url="$1" + case "$url" in + http://*|https://*) return 0 ;; + *) echo "not a valid URL: $url (must start with http:// or https://)" >&2; return 1 ;; + esac +} + +# yt_echo_cmd <args...> +# Prints the yt-dlp command for --dry-run mode +yt_echo_cmd() { + echo "yt-dlp $*" +} + +# classify_url <url> +# Domain classification: music.youtube/soundcloud/bandcamp → audio; +# youtube/youtu.be/vimeo/twitch → video; unknown → ${GRAB_DEFAULT:-video} +# Migrated from bin/pos-media-grab +classify_url() { + local url="$1" mode="${GRAB_DEFAULT:-video}" + case "$url" in + *music.youtube.com*) echo "audio" ;; + *soundcloud.com*) echo "audio" ;; + *bandcamp.com*) echo "audio" ;; + *youtube.com*|*youtu.be*) echo "video" ;; + *vimeo.com*) echo "video" ;; + *twitch.tv*) echo "video" ;; + *) echo "$mode" ;; + esac +} diff --git a/tests/README.md b/tests/README.md index 02dd9c1..d1c771d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -54,4 +54,5 @@ silently. | `t-gen-docs-drift.sh` | `make gen` idempotence on a pristine tracked tree (CI drift gate) | | `t-lint-gate.sh` | `make lint` green on the real tree; planted violations are caught and named | | `t-install-version.sh` | install.sh version gate: match→skip, mismatch→proceed, --force bypass, dry-run variant, flag write, numeric comparison | -| `t-share-mountpoint.sh` | share-client `ask_mountpoint` UX: existing/new/declined/rejected paths, confirm gate, mkdir side effects, non-TTY stdin contract, static `n`→`t` guards | \ No newline at end of file +| `t-share-mountpoint.sh` | share-client `ask_mountpoint` UX: existing/new/declined/rejected paths, confirm gate, mkdir side effects, non-TTY stdin contract, static `n`→`t` guards | +| `t-pos-media-yt.sh` | unified `pos media yt` suite: dispatcher + forwarder resolution, shared yt-lib helpers, yt-mp3/mp4/grab/subtitles flags, dry-run deps, `YT_OUT_DIR` seam, `GRAB_DEFAULT` config, negative controls (unsafe-URL no-expansion, `--lang en,ar` single arg, txt timestamp-stripping) | \ No newline at end of file diff --git a/tests/t-config-precedence.sh b/tests/t-config-precedence.sh index 36f0562..d4fcb06 100644 --- a/tests/t-config-precedence.sh +++ b/tests/t-config-precedence.sh @@ -201,7 +201,7 @@ STUB for tool in pos-ai pos-ai-hf pos-ai-server pos-communication-matrix-listener \ pos-communication-matrix-sender pos-communication-scrcpy \ pos-communication-telegram-listener pos-communication-telegram-sender \ - pos-media-grab pos-network-download; do + pos-media-yt-grab pos-network-download; do if grep -q "load_env_file" "$ROOT/bin/$tool"; then printf ' PASS %s uses shared load_env_file\n' "$tool" else diff --git a/tests/t-pos-media-yt.sh b/tests/t-pos-media-yt.sh new file mode 100755 index 0000000..37fcf7f --- /dev/null +++ b/tests/t-pos-media-yt.sh @@ -0,0 +1,279 @@ +#!/usr/bin/env bash +set -euo pipefail +# t-pos-media-yt.sh — POS--9 verification for the unified `pos media yt` +# suite and its forwarders. +# +# Part 1: dispatcher + forwarder resolution (bin/pos-media-yt, pos-media-mp3/ +# mp4/grab forwarders, full chain through bin/pos) +# Part 2: lib/yt-lib.sh shared helpers (deps, url validation, classify) +# Part 3: yt-mp3 behaviors (flags, dry-run, YT_OUT_DIR seam, unsafe-URL guard) +# Part 4: yt-mp4 behaviors (format select flags, mutual exclusions, seam) +# Part 5: yt-grab classification + delegation (dry-run command strings) + config +# Part 6: yt-subtitles (defaults, --lang en,ar single-arg, formats), +# txt conversion (timestamps stripped), unavailable-subs detection +# Part 7: negative controls mandated by the Architect contract + +run_test() { + local sandbox stubs cfg + sandbox="$(mksandbox media-yt)" + stubs="$sandbox/stubs" + cfg="$sandbox/cfg" + mkdir -p "$stubs" "$cfg" + + local ytdlp_log="$sandbox/ytdlp.log" + : > "$ytdlp_log" + # fake yt-dlp logs its exact argv (one per line) — lets us assert arg + # counting / no-expansion WITHOUT any network or real binary. + cat > "$stubs/yt-dlp" <<STUB +#!/usr/bin/env bash +printf 'yt-dlp %s\\n' "\$*" >> "$ytdlp_log" +STUB + # ffmpeg stub: satisfies the non-dry-run deps guard; not actually used. + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/ffmpeg" + chmod +x "$stubs/yt-dlp" "$stubs/ffmpeg" + + local RB="$ROOT/bin" + local yt ytd="$RB/pos-media-yt" mp3="$RB/pos-media-yt-mp3" \ + mp4="$RB/pos-media-yt-mp4" grab="$RB/pos-media-yt-grab" \ + subs="$RB/pos-media-yt-subtitles" + local env_base=(PATH="$stubs:$RB:/usr/bin:/bin" CONFIG_DIR="$cfg" \ + YT_OUT_DIR="$sandbox/out") + + # ═══ Part 1: dispatcher + forwarders ═══ + # 1.0 bare dispatcher → usage listing subcommands (exit 0) + test_run_env "${env_base[@]}" -- "$ytd" + check_rc "dispatcher bare invocation = usage exit 0" 0 "$TR_RC" + check_contains "dispatcher usage lists mp3" "mp3" "$TR_OUT" + check_contains "dispatcher usage lists subtitles" "subtitles" "$TR_OUT" + + # 1.1 dispatcher unknown subcommand → hard error, nonzero + test_run_env "${env_base[@]}" -- "$ytd" bogus + [ "$TR_RC" -ne 0 ] && printf ' PASS dispatcher unknown subcommand exits nonzero\n' \ + || printf ' FAIL dispatcher unknown subcommand exited 0\n' + check_contains "dispatcher unknown subcommand message" "unknown yt command" "$TR_OUT" + + # 1.2 forwarder helpers: --help passes through to the real yt usage + test_run_env "${env_base[@]}" -- "$RB/pos-media-mp3" --help + check_rc "pos media mp3 --help forwards cleanly" 0 "$TR_RC" + check_contains "mp3 forwarder shows yt-mp3 usage" "pos media yt mp3 [options]" "$TR_OUT" + + test_run_env "${env_base[@]}" -- "$RB/pos-media-mp4" --help + check_rc "pos media mp4 --help forwards cleanly" 0 "$TR_RC" + check_contains "mp4 forwarder shows yt-mp4 usage" "pos media yt mp4 [options]" "$TR_OUT" + + test_run_env "${env_base[@]}" -- "$RB/pos-media-grab" --help + check_rc "pos media grab --help forwards cleanly" 0 "$TR_RC" + check_contains "grab forwarder shows yt-grab usage" "pos media yt grab [options]" "$TR_OUT" + + # 1.3 ytsync forwarder --help forwards to pos media ytsync + test_run_env "${env_base[@]}" -- "$RB/pos-media-yt-ytsync" --help + check_contains "ytsync forwarder references media ytsync" "media ytsync" "$TR_OUT" + + # 1.4 full dispatch chain: `pos media mp3 <url>` → yt-mp3 (fake yt-dlp called). + # `media-yt-mp3` is non-interactive so bin/pos execs it without a tee. + : > "$ytdlp_log" + test_run_env "${env_base[@]}" -- "$RB/pos" media mp3 "http://example.com/x" + check_rc "chain pos media mp3 <url> resolves" 0 "$TR_RC" + check_contains "chain reaches yt-dlp with -x (audio path)" "-x" "$(cat "$ytdlp_log")" + + # ═══ Part 2: lib/yt-lib.sh helpers ═══ + source "$ROOT/lib/common.sh" + source "$ROOT/lib/yt-lib.sh" + # 2.0 yt_validate_url accepts http(s), rejects junk + yt_validate_url "https://youtube.com/watch?v=x" && printf ' PASS url valid http/https accepted\n' \ + || printf ' FAIL valid https url rejected\n' + test_run yt_validate_url "ftp://bad" + [ "$TR_RC" -ne 0 ] && printf ' PASS invalid url rejected\n' || printf ' FAIL invalid url accepted\n' + check_contains "invalid url error message" "not a valid URL" "$TR_OUT" + + # 2.1 classify_url: audio / video / unknown-with-GRAB_DEFAULT + check_eq "music.youtube → audio" "audio" "$(classify_url "https://music.youtube.com/watch?v=x")" + check_eq "soundcloud → audio" "audio" "$(classify_url "https://soundcloud.com/a/b")" + check_eq "youtube.com → video" "video" "$(classify_url "https://youtube.com/watch?v=x")" + check_eq "vimeo → video" "video" "$(classify_url "https://vimeo.com/123")" + check_eq "unknown → default video" "video" "$(classify_url "https://example.com/x")" + check_eq "unknown → GRAB_DEFAULT=audio" "audio" "$(GRAB_DEFAULT=audio classify_url "https://example.com/x")" + + # ═══ Part 3: yt-mp3 ═══ + # 3.0 dry-run requires no deps and prints the yt-dlp command (audio flags) + test_run_env "${env_base[@]}" -- "$mp3" --dry-run "https://youtube.com/watch?v=dQw4w9WgXcQ" + check_rc "mp3 dry-run ok" 0 "$TR_RC" + check_contains "mp3 dry-run has -x --audio-format mp3" "--audio-format mp3" "$TR_OUT" + check_contains "mp3 dry-run has --embed-metadata" "--embed-metadata" "$TR_OUT" + + # 3.1 --by-artist template + test_run_env "${env_base[@]}" -- "$mp3" --dry-run --by-artist "https://youtu.be/x" + check_contains "mp3 --by-artist organizes by artist/uploader" "%(artist,uploader)s/" "$TR_OUT" + + # 3.2 --no-playlist propagated + test_run_env "${env_base[@]}" -- "$mp3" --dry-run --no-playlist "https://youtube.com/watch?v=x" + check_contains "mp3 --no-playlist passed through" "--no-playlist" "$TR_OUT" + + # 3.3 --cookies must exist + test_run_env "${env_base[@]}" -- "$mp3" --dry-run --cookies "$sandbox/nope.txt" "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS mp3 missing cookies file errors\n' \ + || printf ' FAIL mp3 accepted missing cookies file\n' + + # 3.4 YT_OUT_DIR seam overrides default output + test_run_env "${env_base[@]}" -- "$mp3" --dry-run "https://youtube.com/watch?v=x" + check_contains "mp3 uses YT_OUT_DIR seam" "$sandbox/out/%(title)s.%(ext)s" "$TR_OUT" + + # 3.5 missing dep in non-dry-run → hard error (no yt-dlp on PATH at all) + test_run_env PATH="/usr/bin:/bin" -- "$mp3" "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS mp3 without yt-dlp in path errors\n' \ + || printf ' FAIL mp3 ran without yt-dlp present\n' + + # 3.6 no URL → usage exit 0 + test_run_env "${env_base[@]}" -- "$mp3" --dry-run + check_rc "mp3 no URL → usage exit 0" 0 "$TR_RC" + check_contains "mp3 no URL prints usage" "Usage: pos media yt mp3" "$TR_OUT" + + # 3.7 NEGATIVE CONTROL: unsafe URL not expanded — passed as ONE literal arg. + # `$(echo pwned)` stays literal inside the URL token; no second pwned word + # appears in the argv log (which would mean command substitution ran). + : > "$ytdlp_log" + bad='$(echo pwned)' + test_run_env "${env_base[@]}" -- "$mp3" "http://example.com/$bad" + check_rc "unsafe URL run (fake yt-dlp) ok" 0 "$TR_RC" + local logged; logged="$(cat "$ytdlp_log")" + check_contains "url passed literally (single arg)" "\$(echo pwned)" "$logged" + check_not_contains "no separate pwned word from substitution" "example.com/ pwned" "$logged" + + # ═══ Part 4: yt-mp4 ═══ + # 4.0 --best + test_run_env "${env_base[@]}" -- "$mp4" --dry-run --best "https://youtube.com/watch?v=x" + check_contains "mp4 --best selects bestvideo+bestaudio" "bestvideo*+bestaudio/best" "$TR_OUT" + + # 4.1 --worst + test_run_env "${env_base[@]}" -- "$mp4" --dry-run --worst "https://youtube.com/watch?v=x" + check_contains "mp4 --worst selects worst" "-f worst" "$TR_OUT" + + # 4.2 -f id skips prompt + test_run_env "${env_base[@]}" -- "$mp4" --dry-run -f 22 "https://youtube.com/watch?v=x" + check_contains "mp4 -f 22 (no prompt)" "-f 22" "$TR_OUT" + + # 4.3 mutual exclusions + test_run_env "${env_base[@]}" -- "$mp4" --dry-run --best --worst "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS mp4 --best/--worst mutually exclusive\n' \ + || printf ' FAIL mp4 allowed --best and --worst together\n' + test_run_env "${env_base[@]}" -- "$mp4" --dry-run -f 22 --best "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS mp4 --format/--best mutually exclusive\n' \ + || printf ' FAIL mp4 allowed --format and --best together\n' + + # 4.4 YT_OUT_DIR seam + test_run_env "${env_base[@]}" -- "$mp4" --dry-run --best "https://youtube.com/watch?v=x" + check_contains "mp4 uses YT_OUT_DIR seam" "$sandbox/out/%(title)s.%(ext)s" "$TR_OUT" + + # 4.5 no URL → usage + test_run_env "${env_base[@]}" -- "$mp4" --dry-run --best + check_rc "mp4 no URL → usage exit 0" 0 "$TR_RC" + check_contains "mp4 no URL prints usage" "Usage: pos media yt mp4" "$TR_OUT" + + # ═══ Part 5: yt-grab classification + delegation (dry-run) + config ═══ + # 5.0 music.youtube → audio route (dry-run prints delegate command, no exec) + test_run_env "${env_base[@]}" -- "$grab" --dry-run "https://music.youtube.com/watch?v=abc" + check_contains "grab music.youtube routes to yt mp3" "pos media yt mp3" "$TR_OUT" + check_not_contains "grab music.youtube not routed to mp4" "pos media yt mp4" "$TR_OUT" + + # 5.1 youtube.com → video route (default --best) + test_run_env "${env_base[@]}" -- "$grab" --dry-run "https://youtube.com/watch?v=xyz" + check_contains "grab youtube routes to yt mp4" "pos media yt mp4 --best" "$TR_OUT" + + # 5.2 --audio forces audio despite video domain + test_run_env "${env_base[@]}" -- "$grab" --dry-run --audio "https://youtube.com/watch?v=abc" + check_contains "grab --audio forces yt mp3" "pos media yt mp3" "$TR_OUT" + + # 5.3 --worst → video --worst + test_run_env "${env_base[@]}" -- "$grab" --dry-run --worst "https://youtube.com/watch?v=abc" + check_contains "grab --worst routes to mp4 --worst" "pos media yt mp4 --worst" "$TR_OUT" + + # 5.4 --audio + --video mutually exclusive + test_run_env "${env_base[@]}" -- "$grab" --dry-run --audio --video "https://youtube.com/watch?v=abc" + [ "$TR_RC" -ne 0 ] && printf ' PASS grab --audio/--video mutually exclusive\n' \ + || printf ' FAIL grab allowed --audio and --video together\n' + + # 5.5 invalid URL + test_run_env "${env_base[@]}" -- "$grab" "not-a-url" + [ "$TR_RC" -ne 0 ] && printf ' PASS grab rejects invalid url\n' \ + || printf ' FAIL grab accepted invalid url\n' + check_contains "grab error prefixed yt-grab" "yt-grab:" "$TR_OUT" + + # 5.6 unknown domain honors GRAB_DEFAULT via config file (audio) + printf 'GRAB_DEFAULT=audio\n' > "$cfg/grab.env" + test_run_env "${env_base[@]}" -- "$grab" --dry-run "https://example.com/x" + check_contains "grab unknown domain honors GRAB_DEFAULT=audio" "pos media yt mp3" "$TR_OUT" + + # ═══ Part 6: yt-subtitles ═══ + # 6.0 defaults: --write-subs --write-auto-subs --sub-langs best + test_run_env "${env_base[@]}" -- "$subs" --dry-run "https://youtube.com/watch?v=x" + check_contains "subs default writes manual subs" "--write-subs" "$TR_OUT" + check_contains "subs default writes auto subs" "--write-auto-subs" "$TR_OUT" + check_contains "subs default sub-langs best" "--sub-langs best" "$TR_OUT" + + # 6.1 NEGATIVE CONTROL: --lang en,ar becomes ONE --sub-langs arg + test_run_env "${env_base[@]}" -- "$subs" --dry-run --lang en,ar "https://youtube.com/watch?v=x" + check_contains "subs --lang en,ar single arg" "--sub-langs en,ar" "$TR_OUT" + check_not_contains "subs does not split --lang (no single 'en')" "--sub-langs en " "$TR_OUT" + + # 6.2 --format vtt + test_run_env "${env_base[@]}" -- "$subs" --dry-run --format vtt "https://youtube.com/watch?v=x" + check_contains "subs --format vtt" "--sub-format vtt" "$TR_OUT" + + # 6.3 --auto-only drops --write-subs + test_run_env "${env_base[@]}" -- "$subs" --dry-run --auto-only "https://youtube.com/watch?v=x" + check_not_contains "subs --auto-only drops manual subs" "--write-subs" "$TR_OUT" + check_contains "subs --auto-only keeps auto subs" "--write-auto-subs" "$TR_OUT" + + # 6.4 --output dir template + test_run_env "${env_base[@]}" -- "$subs" --dry-run --output "$sandbox/subs" "https://youtube.com/watch?v=x" + check_contains "subs --output template" "$sandbox/subs/%(title)s.%(sub_lang)s.%(ext)s" "$TR_OUT" + + # 6.5 invalid --format + test_run_env "${env_base[@]}" -- "$subs" --dry-run --format bogus "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS subs rejects unknown format\n' \ + || printf ' FAIL subs accepted unknown format\n' + + # 6.6 unavailable subs: fake yt-dlp fails with no-subtitle stderr → friendly error + mkdir -p "$stubs/subsfail" + cat > "$stubs/subsfail/yt-dlp" <<STUB +#!/usr/bin/env bash +printf 'ERROR: no subtitles found' >&2 +exit 1 +STUB + chmod +x "$stubs/subsfail/yt-dlp" + test_run_env PATH="$stubs/subsfail:/usr/bin:/bin" -- "$subs" "https://youtube.com/watch?v=x" + [ "$TR_RC" -ne 0 ] && printf ' PASS subs no-subtitles exits nonzero\n' \ + || printf ' FAIL subs no-subtitles exited 0\n' + check_contains "subs unavailable-subs message" "unavailable subtitles" "$TR_OUT" + + # 6.7 NEGATIVE CONTROL: txt conversion strips SRT timestamps — run the real +# tool non-dry with a fake yt-dlp that writes an .srt into the cwd; the txt +# converter must strip timestamps/html and keep the words. + local txtdir="$sandbox/txtdir" + mkdir -p "$txtdir" "$stubs/txtsubs" + cat > "$stubs/txtsubs/yt-dlp" <<STUB +#!/usr/bin/env bash +printf '1\\n00:00:01,000 --> 00:00:02,000\\nHello <i>world</i>\\n\\n2\\n00:00:03,000 --> 00:00:04,000\\nSecond line\\n' > foo.srt +exit 0 +STUB + chmod +x "$stubs/txtsubs/yt-dlp" + ( + cd "$txtdir" + env PATH="$stubs/txtsubs:/usr/bin:/bin" "$subs" --format txt "https://youtube.com/watch?v=x" + ) >"$sandbox/txtrun.log" 2>&1 + local txt; txt="$(cat "$txtdir/foo.txt" 2>/dev/null || true)" + check_not_contains "txt output strips timestamps" "-->" "$txt" + check_not_contains "txt output strips html tags" "<i>" "$txt" + check_contains "txt output keeps words" "Hello world" "$txt" + + # 6.8 subs dry-run skips deps (yt-dlp not in path) — needs no binaries + test_run_env PATH="/usr/bin:/bin" -- "$subs" --dry-run "https://youtube.com/watch?v=x" + check_rc "subs dry-run works without yt-dlp (skips deps)" 0 "$TR_RC" + check_contains "subs dry-run prints yt-dlp cmd" "yt-dlp" "$TR_OUT" + + # 6.9 subs no URL → error (not usage), exit 1 + test_run_env "${env_base[@]}" -- "$subs" --dry-run + [ "$TR_RC" -ne 0 ] && printf ' PASS subs no URL exits nonzero\n' \ + || printf ' FAIL subs no URL exited 0\n' + check_contains "subs no URL error message" "missing URL (see --help)" "$TR_OUT" +}