Your Name
072a8e72c1
gates / consistency-and-conventions (push) Successful in 2m14s
New tool that classifies URLs by domain and delegates to pos media mp3 (audio) or pos media mp4 (video). Listener gains URL detection step between prefix map and AI bridge — bare URLs auto-download. Domain rules: music.youtube.com/soundcloud/bandcamp → mp3, youtube/vimeo/twitch → mp4 --best, unknown → configurable default. 28 test cases / 70 assertions / 0 failed. make gen && make check green, make lint 0 FAIL / 0 WARN.
308 lines
12 KiB
Markdown
308 lines
12 KiB
Markdown
# How-To: `pos media`
|
||
|
||
Download audio and video from the web via `yt-dlp`, auto-classify URLs,
|
||
sync your library to a USB stick, and keep YouTube channels incrementally
|
||
up to date.
|
||
Tools: `grab`, `mp3`, `mp4`, `sync`, `ytsync`.
|
||
|
||
| Tool | What it does |
|
||
|------|--------------|
|
||
| `pos media grab` | Auto-classify URL and download as audio or video |
|
||
| `pos media mp3` | Download audio, convert to MP3 |
|
||
| `pos media mp4` | Download video with smart/interactive format selection |
|
||
| `pos media sync` | Incrementally copy `~/Music` onto a USB stick (mp3/mp4) |
|
||
| `pos media ytsync` | Track YouTube channels/playlists and download only new videos into `~/Videos` |
|
||
|
||
Requires `yt-dlp` and `ffmpeg` (`sudo apt install yt-dlp ffmpeg`); the tools
|
||
fail with a clean error message instead of a raw `command not found` if either
|
||
is missing.
|
||
|
||
---
|
||
|
||
## `pos media mp3` — audio as MP3
|
||
|
||
```bash
|
||
pos media mp3 <url>
|
||
```
|
||
|
||
Extracts and converts the audio track to MP3 in `~/Music/`. With `--by-artist`
|
||
files land in `~/Music/<artist>/<title>.mp3` (falls back to the uploader name
|
||
when there's no artist tag), so a library stays organized.
|
||
|
||
```bash
|
||
pos media mp3 https://youtube.com/watch?v=dQw4w9WgXcQ
|
||
pos media mp3 --by-artist https://youtu.be/dQw4w9WgXcQ
|
||
```
|
||
|
||
MP3s are encoded at best quality with title/artist/album/date/chapters embedded
|
||
(`--embed-metadata --embed-chapters`, artist falls back to uploader) plus the
|
||
cover art as a JPEG thumbnail. Existing files are never overwritten.
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `-o, --output <dir>` | Output directory (default `~/Music`) |
|
||
| `--by-artist` | Organize as `<dir>/<artist>/<title>.mp3` |
|
||
| `--no-playlist` | Download only the single video, not the playlist |
|
||
| `--cookies <file>` | Netscape cookies.txt for age-gated content |
|
||
| `--dry-run` | Print the yt-dlp command without running it |
|
||
|
||
**Recipe:** batch — loop over a list of URLs:
|
||
```bash
|
||
while read -r url; do pos media mp3 --by-artist "$url"; done < urls.txt
|
||
```
|
||
|
||
---
|
||
|
||
## `pos media mp4` — video with smart format choice
|
||
|
||
```bash
|
||
pos media mp4 <url>
|
||
```
|
||
|
||
Without a format flag, the available formats are shown as a **short curated
|
||
list** (grouped `[audio]` / `[video]` / `[combo]`, with the raw dump's clutter
|
||
dropped) and you pick one — the id is validated before downloading. Entering
|
||
nothing (or `best`) picks the best video+audio automatically.
|
||
|
||
Non-interactive (scripting-friendly):
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `-f, --format <id>` | Download that format id directly (no prompt) |
|
||
| `--best` | Best video + audio, no prompt |
|
||
| `--worst` | Lowest quality, no prompt |
|
||
| `-o, --output <dir>` | Output directory (default `~/Videos`) |
|
||
| `--no-playlist` | Download only the single video |
|
||
| `--cookies <file>` | Netscape cookies.txt for age-gated content |
|
||
| `--dry-run` | Print the yt-dlp command without running it |
|
||
|
||
```bash
|
||
pos media mp4 --best https://youtube.com/watch?v=dQw4w9WgXcQ
|
||
pos media mp4 -f 22 https://youtube.com/watch?v=dQw4w9WgXcQ
|
||
```
|
||
|
||
Videos merge to MP4 with metadata, chapters, subtitles (all languages) and the
|
||
thumbnail embedded (`--embed-metadata --embed-chapters --embed-subs
|
||
--sub-langs all --embed-thumbnail`); existing files are never overwritten.
|
||
|
||
**Recipe:** grab a 4K stream for later — `--best` already picks the best
|
||
video+audio and merges them.
|
||
|
||
---
|
||
|
||
## `pos media grab` — auto-classify URL and download
|
||
|
||
```bash
|
||
pos media grab <url>
|
||
```
|
||
|
||
Smart URL classifier that routes to `pos media mp3` or `pos media mp4`
|
||
automatically based on the domain. Send a URL from your phone via Telegram and
|
||
the bot downloads it to the right place without you thinking about it.
|
||
|
||
**Classification rules:**
|
||
|
||
| Domain | Routes to | Why |
|
||
|--------|-----------|-----|
|
||
| `music.youtube.com` | mp3 | Audio streaming |
|
||
| `soundcloud.com` | mp3 | Audio-first platform |
|
||
| `bandcamp.com` | mp3 | Audio-first platform |
|
||
| `youtube.com` / `youtu.be` | mp4 | Video content |
|
||
| `vimeo.com` / `twitch.tv` | mp4 | Video platforms |
|
||
| Everything else | mp4 (default) | Safe fallback |
|
||
|
||
Override the classification with `--audio` or `--video`. The default for
|
||
unrecognized domains is `video` — change it with `pos config grab` or set
|
||
`GRAB_DEFAULT=audio` in `~/.config/linux_post_install/grab.env`.
|
||
|
||
```bash
|
||
pos media grab https://music.youtube.com/watch?v=abc # → ~/Music
|
||
pos media grab https://youtube.com/watch?v=xyz # → ~/Videos
|
||
pos media grab --audio https://vimeo.com/123 # force mp3
|
||
pos media grab --worst https://youtu.be/abc # lowest quality
|
||
pos media grab --dry-run https://soundcloud.com/artist/track # preview only
|
||
```
|
||
|
||
Non-interactive by design — `pos media mp4` receives `--best` by default so it
|
||
never prompts for a format (critical for Telegram bot context where there's no
|
||
TTY). Pass `--worst` if you want the smallest file.
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `--audio` | Force audio (mp3) download |
|
||
| `--video` | Force video (mp4) download |
|
||
| `--best` | Best quality for video (default) |
|
||
| `--worst` | Lowest quality for video |
|
||
| `-o, --output <dir>` | Output directory (passed to mp3/mp4) |
|
||
| `--no-playlist` | Download only the single video |
|
||
| `--cookies <file>` | Netscape cookies.txt for age-gated content |
|
||
| `--dry-run` | Print the command that would run, don't execute |
|
||
|
||
---
|
||
|
||
## `pos media sync` — music onto a USB stick
|
||
|
||
```bash
|
||
pos media sync # copy everything (mp3 + mp4) from ~/Music to the stick
|
||
pos media sync --mp3 # only the .mp3 files
|
||
pos media sync --mp4 # only the .mp4 files
|
||
```
|
||
|
||
Detects connected USB storage exactly like `pos system backup` (same shared
|
||
`lib/usb-lib.sh`): a plugged-in but unmounted stick is offered a mount first
|
||
(`/media/<label>`, world-writable, mirrors `usb-automount`), multiple sticks
|
||
are listed for you to pick, and if nothing is plugged in it re-scans after you
|
||
press Enter. Files are mirrored into `<usb>/Music/` (change with
|
||
`MEDIA_SYNC_DEST`) preserving the artist/album tree.
|
||
|
||
**Sync semantics — add/update only, never delete.** Files missing on the stick
|
||
are copied; files whose size or mtime changed are overwritten; everything
|
||
identical is skipped. Files on the stick that are no longer in the source are
|
||
**left alone** — a playback stick can never lose files to a mirror mistake.
|
||
Copies keep the source timestamps (`cp --preserve=timestamps`), so a re-run is
|
||
a no-op. Preview before copying with `--dry-run`:
|
||
|
||
```bash
|
||
pos media sync --mp4 --dry-run # shows "would copy" list + counts, copies nothing
|
||
```
|
||
|
||
The source folder may be a symlink to a library elsewhere
|
||
(`~/Music -> /mnt/data/music`) — it is followed, the artist/album tree is
|
||
mirrored under the symlink's target.
|
||
|
||
**Target picking.** Mounted USB partitions are listed with size, label and
|
||
filesystem (single candidate → confirm prompt; several → numbered picker, one
|
||
row per partition). EFI system partitions (e.g. a Ventoy stick's `VTOYEFI`,
|
||
32 MB) are **never** offered as a target — they are boot machinery, not
|
||
storage. Before any copy the tool verifies the payload fits (`df` vs the
|
||
exact bytes to copy) and fails fast with `Not enough free space on …` instead
|
||
of dying mid-copy with `No space left on device`.
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `--mp3` | Sync only `*.mp3` (neither flag = both) |
|
||
| `--mp4` | Sync only `*.mp4` (neither flag = both) |
|
||
| `--source <dir>` | Source folder (default `~/Music`) |
|
||
| `--dry-run` | Preview what would be copied, copy nothing |
|
||
|
||
When it finishes it announces the result via `lib/notify.sh`
|
||
(`Music sync completed: N added, M updated → <usb>/Music`).
|
||
|
||
**Recipes:**
|
||
```bash
|
||
pos media sync # keep the car stick up to date (both formats)
|
||
pos media sync --mp3 --dry-run # check what a new batch will bring first
|
||
pos media sync --source /data/Music # sync a library that lives elsewhere
|
||
```
|
||
|
||
Config (all in `~/.config/linux_post_install/system.env` or exported):
|
||
`MEDIA_SYNC_SOURCE` (default `$HOME/Music`), `MEDIA_SYNC_DEST` (default
|
||
`Music`), plus the shared `USB_MOUNT_BASE` / `USB_BYID` seams.
|
||
|
||
---
|
||
|
||
## `pos media ytsync` — incremental YouTube channel sync
|
||
|
||
```bash
|
||
pos media ytsync # interactive
|
||
pos media ytsync add https://youtube.com/@SomeChannel
|
||
pos media ytsync sync # fetch new videos from every source
|
||
pos media ytsync list
|
||
pos media ytsync remove "Some Channel"
|
||
```
|
||
|
||
The first run asks for a channel or playlist URL, shows what was resolved
|
||
(`Resolved : Linus Tech Tips (channel · 2140 videos)` + the target library path),
|
||
and downloads everything. Every later run probes the same URL, diffs it against a
|
||
per-source download archive, and fetches **only new videos** — one yt-dlp call per
|
||
video (`bestvideo*+bestaudio/best` merged to MP4 with metadata, chapters and the
|
||
thumbnail embedded; existing files are never overwritten).
|
||
|
||
Where files land:
|
||
|
||
```
|
||
~/Videos/<channel>/<title>.mp4 # channels & single videos
|
||
~/Videos/<channel>/<playlist>/<NNN> - <title>.mp4 # playlist sources (playlist order)
|
||
```
|
||
|
||
| Command | Meaning |
|
||
|---------|---------|
|
||
| `add [url]` | Register a source + first download (asks for the URL if omitted) |
|
||
| `sync [name]` | Incremental pass — all tracked sources, or one by slug/display name |
|
||
| `list` | Tracked sources table: type, archived count, last sync date, destination |
|
||
| `remove <name>` | Stop tracking. Downloaded files AND the archive are kept |
|
||
|
||
Notes:
|
||
|
||
- A watch link with **both** `?v=` and `&list=` downloads only that single video,
|
||
never the whole playlist; `youtu.be/<id>` short links count as watch links too.
|
||
- Retitled/renamed videos keep their local filenames (the archive is keyed by video
|
||
id); inserting a video mid-playlist shifts future numbering only.
|
||
- Members-only / age-gated videos are skipped with a count ("N videos require
|
||
sign-in"). Escape hatch: put `YTSYNC_EXTRA_ARGS=--cookies-from-browser firefox`
|
||
(or `--cookies <file>`) into the config file below.
|
||
- State lives outside `~/Videos`, in `~/.local/share/linux_post_install/ytsync/`
|
||
(registry, per-source archives, run history) — the media tree stays pure media.
|
||
|
||
**Recipe: daily automation** — no built-in timers; use the shared scheduler:
|
||
|
||
```bash
|
||
pos system schedule config # name: ytsync
|
||
# INTERVAL=daily (or hourly / weekly / OnCalendar=…)
|
||
# NOTIFY=never # ytsync sends its own digest; don't double-notify
|
||
# MSG="ytSync"
|
||
# COMMAND=pos media ytsync sync
|
||
pos system schedule enable ytsync
|
||
pos system schedule run ytsync # test once, right now
|
||
```
|
||
|
||
ytsync never prompts on `sync`, so the job is timer-safe by construction. It
|
||
notifies via `lib/notify.sh` only when something happened (new videos or failures);
|
||
a scheduled no-op run stays silent — hence `NOTIFY=never` on the job to avoid
|
||
double alerts.
|
||
|
||
**Previewing:** `--dry-run` works on `add` and `sync` — real probe + plan block
|
||
(`New : 12 would be downloaded (800 already present)` + example filenames), zero
|
||
writes anywhere:
|
||
|
||
```bash
|
||
pos media ytsync sync --dry-run
|
||
pos media ytsync add https://youtube.com/@SomeChannel --dry-run
|
||
```
|
||
|
||
Config (`~/.config/linux_post_install/ytsync.env`, chmod 600 — materialize/edit via
|
||
`pos config ytsync`; exported environment wins):
|
||
|
||
| Key | Default | Meaning |
|
||
|-----|---------|---------|
|
||
| `YTSYNC_VIDEOS_DIR` | `$HOME/Videos` | Videos root for synced sources |
|
||
| `YTSYNC_EXTRA_ARGS` | *(empty)* | Extra yt-dlp flags appended to every download call (cookies, format overrides, …) |
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
- Format list is empty / download fails → the site or age-gate requires
|
||
cookies; pass `--cookies ~/cookies.txt` (export it from your browser), or
|
||
update yt-dlp (`sudo apt upgrade yt-dlp`).
|
||
- Error about a missing postprocessor → `sudo apt install ffmpeg`.
|
||
- `--by-artist` leaves files loose → the source has no artist/uploader tag;
|
||
it falls back to the uploader name in the artist slot.
|
||
- Very large downloads: ensure free space; files land in `~/Music`/`~/Videos`
|
||
(or your `-o` directory).
|
||
- ytsync: "could not reach YouTube" at add → connectivity/DNS; the interactive
|
||
flow re-prompts up to 3×. "source not found or private" → wrong handle or a
|
||
deleted/private source.
|
||
- ytsync: many "exists, kept" warnings after a YouTube-side change → the
|
||
channel's internal id changed, so re-adding computed a new archive. Rename the
|
||
old archive file (`~/.local/share/linux_post_install/ytsync/archive/<old>.txt`)
|
||
to the new slug before syncing (see `tools-docs/ytsync.md`).
|
||
- ytsync: "disk full — stopping … mid-run" → free space on the videos volume and
|
||
run `pos media ytsync sync` again; unfetched videos simply stay "new".
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- Reference: [DOC/POS.md → media](../POS.md)
|