Linux_post_install/tools-docs/ytsync.md

pos media ytsync — implementation research notes

Internal reference for maintainers touching bin/pos-media-ytsync. The user-facing docs live in DOC/POS.md → media and DOC/howto/media.md; the behavioral contract is reportAgents/2026-08-22-designer-ytsync.md + the Architect decisions D1–D9 in reportAgents/2026-08-22-architect-ytsync.md.

Probe mechanics

One fast call per source per run:

yt-dlp --flat-playlist -J --no-warnings -- <url>

• --flat-playlist keeps entries as stubs, so cost is 1–2 HTTP round trips regardless of library size (seconds even for 2000-video channels).
• -J dumps a single JSON object. Parsed with jq:
  • display name: .channel // .uploader // .uploader_id // .title
  • owner (for playlist subdirs): .channel // .uploader // .uploader_id — no title fallback (avoids Playlist Title/Playlist Title doubling when a playlist probe has no channel fields)
  • key (slug source): .uploader_id // .channel_id // .id
  • entries: .entries[] | (.id) + "\x1f" + (.title); a ?v= URL returns a bare video object with no .entries, handled as a one-entry list.
• The new-list is computed BEFORE any download by diffing entry ids against archive/<slug>.txt — exact [n/N] counts, exact dry-run plans, zero speculative downloads.
• Sign-in-skipped bucket: entries with an empty id OR titles starting with [Private / [Deleted / [Unavailable. Reported as "N videos require sign-in — skipped"; never counted as failures.

Why probes don't use spawn()

lib/common.sh spawn() exits the process on failure and captures output until completion. ytsync needs recoverable probes (interactive add re-prompts ≤3×; sync must continue other sources after one bad probe) and never wraps the download batch (hours of silence behind a spinner). _probe_run mirrors spawn's OK/FAIL line UX but returns rc, and its braille spinner frames are emitted only on a TTY so no \r bytes ever reach the dispatcher's tee'd logs.

Classification (URL shape, deterministic)

URL contains	Type	Behavior
youtu.be/<id> path (with or without list=)	video	short links name one video — treated exactly like v=
v= present (with or without list=)	video	single video, downloaded with --no-playlist; nobody backfills a 500-video playlist by pasting a watch link
list= without any video-naming part	playlist	tracked source; files land as <videos>/<owner>/<playlist>/<NNN> - <title>.<ext>
neither	channel	flat source; <videos>/<channel>/<title>.<ext>

Download invocation (one yt-dlp call PER NEW VIDEO)

yt-dlp -f "bestvideo*+bestaudio/best"
       --merge-output-format mp4        # parity with pos media mp4
       --embed-metadata --embed-chapters --embed-thumbnail
       --convert-thumbnails jpg
       --no-overwrites                  # parity; collisions become "exists, kept" warnings
       --download-archive <state>/archive/<slug>.txt   # crash-safe per-video recording
       --windows-filenames              # USB/Samba/TV-safe names
       --trim-filenames 120             # headroom for the NNN prefix under 255-byte limits
       --retries 3 --fragment-retries 3
       [--no-playlist]                  # only for type=video sources
       ${YTSYNC_EXTRA_ARGS}             # appended last — user override hatch
       -o "<template>" "https://www.youtube.com/watch?v=<id>"

• Templates are COMPUTED from stored registry fields (subdir), never from yt-dlp placeholders like %(playlist_title)s — a stored subdir cannot be NA and cannot drift mid-library.
• Playlist template gets a LITERAL zero-padded index injected by the tool (from the probe's entry position), because standalone watch URLs have no live %(playlist_index)s. Caveat: inserting a video mid-playlist shifts FUTURE numbering; existing files are never renamed.
• Deliberate divergences from mp4: no --embed-subs --sub-langs all (library bloat; re-add per-source via YTSYNC_EXTRA_ARGS, appended-last wins); --quiet --no-warnings plus --progress only on a TTY (yt-dlp auto-simplifies progress when piped, keeping \r out of logs).
• Sequential downloads; batching is the listed future optimization (accepted v1 cost: one extra extraction round trip per video).

State layout (machine-owned, outside ~/Videos)

${YTSYNC_STATE_DIR:-~/.local/share/linux_post_install/ytsync}/
├── registry            # \x1f-delimited: slug⇥type⇥url⇥subdir⇥playlist_title⇥added_ts
├── archive/<slug>.txt  # native yt-dlp archive format ("<extractor> <id>"), one per source
└── history.log         # append-only: "<YYYY-MM-DD HH:MM> · <name> · N new · M skipped · K failed"

• slug: [a-z0-9][a-z0-9_-]*, derived from probed uploader_id/handle; numeric -2 suffix on collision, checking registry slugs only.
• Registry/archive writes are atomic (mktemp inside the state dir + mv).
• remove drops only the registry line. Keeping the orphaned archive makes a future re-add of the same source an incremental resume instead of a full re-download that would collide with existing files under --no-overwrites.
• History grammar note: a wholesale probe failure appends <date> · <name> · FAILED (probe) instead of the numeric triple — the pass did not complete, and faking zeros would hide it from grep.

Edge cases

• Mid-playlist inserts shift future numbering only (see above).
• Title renames on YouTube never rename local files — the archive is keyed by video id; local files are immutable once written.
• Same-title collision (--no-overwrites refusal): stderr is matched for "already been downloaded" → [!] exists, kept: <file>, counted into the "already present" tally of the summary (the fixed summary grammar has three buckets; tools-docs records this folding).
• Disk full (ENOSPC): stderr matched for "No space left on device"/Errno 28 → prominent warning, that source stops mid-run (unfetched videos simply stay "new" next run since the archive was untouched), other channels continue.
• Per-video generic failure: [!] unavailable: <title> + first 3 stderr lines, continue, counted as failed. Per-video failures never flip the exit code; rc 1 is reserved for missing deps, invalid explicit URL, unknown/ambiguous <name>, or ≥1 requested source failing wholesale during sync.
• Slug stability: the slug derives from the probed uploader_id. If YouTube ever changes that id for a channel, a re-add computes a different slug whose archive starts empty → existing files would be "re-downloaded", hitting --no-overwrites and spamming exists, kept warnings. Manual mitigation if it ever bites: rename the old archive/<old-slug>.txt to the new slug before syncing. Accepted v1 limitation (Architect §Remaining uncertainty #4).

Testing seams

• YTSYNC_STATE_DIR, YTSYNC_VIDEOS_DIR — every written path honors them (written VAR="${VAR:-default}"). Fake yt-dlp arrives via stub PATH emitting canned -J JSON; fake notify via a stubbed sender appending to sends.log.
• Interactive reads all come from /dev/tty (NOT stdin), so the tool is NOT in bin/pos's INTERACTIVE_CMDS and dispatched runs keep full tee logging; non-tty interactive entry points print the guard line and exit 0.