Linux_post_install/AgentsReport/reviewer/2026-09-04_hf-downloader-review.md

# Reviewer Report — `pos ai hf` (Hugging Face Model Downloader)

**Date:** 2026-09-04
**Status:** CHANGES_REQUIRED

---

## TL;DR

**Verdict: CHANGES_REQUIRED**

Reviewed: `bin/pos-ai-hf` (470 lines), `DOC/POS.md` updates, generated docs/completions, AGENT_TODO entry.

**1 BLOCKING finding** — `total_size` accumulated inside a pipe subshell is always 0, so multi-file download summaries show incorrect total size. **2 REQUIRED findings** — missing disk space pre-flight check (architect-specified) and missing `hf_repo_files` API fallback (architect-specified). **2 SUGGESTED findings** — rate-limit HEAD request inefficiency and error message format deviation.

| Severity | Count |
|----------|-------|
| BLOCKING | 1 |
| REQUIRED | 2 |
| SUGGESTED | 2 |
| NOTE | 3 |

---

## Checklist Results

### Code Quality

| Item | Status | Evidence |
|------|--------|----------|
| `set -euo pipefail` present | [PASS] | Line 2: `set -euo pipefail` |
| `# POS:` header correct format with em-dash | [PASS] | Line 3: `# POS: ai hf — Download AI models from Hugging Face (search, download, manage)` — em-dash `—` confirmed |
| `# POS_FLAGS:` correct | [PASS] | Line 4: `# POS_FLAGS: --branch --gguf --output` — matches actual flag parsing (lines 88-109) |
| `# POS_DEPS:` correct | [PASS] | Line 5: `# POS_DEPS: curl jq` — matches deps guards on lines 17-18 |
| `# POS_CONFIG:` correct format | [PASS] | Line 6: `# POS_CONFIG: ai \| ai.env \| HF_TOKEN=secret:… \| HF_DOWNLOAD_DIR=:…` — uses `secret:` prefix convention matching other tools (pos-docker-compose, pos-communication-matrix-sender, pos-communication-telegram-sender) |
| `# POS_EXAMPLES:` present | [PASS] | Lines 7-12: 6 example lines covering search, download (repo, gguf, single file), list, remove |
| Sources `lib/common.sh` via standard fallback | [PASS] | Line 14: `source "$(dirname "$0")/../lib/common.sh" 2>/dev/null \|\| source "$(dirname "$0")/common.sh"` — exact template pattern |
| Deps guards BEFORE `-h\|--help` | [PASS] | Lines 17-18 (deps) before line 90 (`-h\|--help` case) |
| `usage()` present and comprehensive | [PASS] | Lines 44-79: all 4 subcommands, download options, examples, config keys, exit codes documented |
| All 4 subcommands implemented | [PASS] | `cmd_search` (line 279), `cmd_download` (line 303), `cmd_list` (line 402), `cmd_remove` (line 438); dispatch at line 464 |
| Config loader reads `ai.env` with env-var precedence | [PASS] | `load_hf_config()` (lines 25-39): reads `CONFIG_FILE`, env-already-set wins (`if [ -z "${!k:-}" ]`), strips quotes, CR, comments |
| Auth header: `Authorization: Bearer $HF_TOKEN` | [PASS] | `hf_auth_header()` (lines 128-132): `printf 'Authorization: Bearer %s' "$HF_TOKEN"` |
| Rate limit handling: 429 → sleep + retry | [PASS] | Lines 150-167: loop with `attempt < 2`, on 429 extracts `Retry-After` or defaults 60, sleeps, retries once |
| Resume: `curl -C -` | [PASS] | Line 259: `curl_args=(-L -C - --progress-bar -o "$target")` |
| `.hf-meta` metadata tracking | [PASS] | Lines 368-382: writes JSON with repo_id, branch, timestamp, files array |
| Output matches contract (emojis, paths, sizes) | [PASS] | Lines 392-398: 📥 and 📁 emojis, repo-id, size, path format matches Architect Decision 6 |
| Error handling: 404 | [PASS] | Line 175: `err "Model not found: ${endpoint#/api/models/}"` |
| Error handling: 401/403 | [PASS] | Line 174: `err "Authentication failed — check HF_TOKEN (pos config ai)"` |
| Error handling: 429 | [PASS] | Line 176: `err "Rate limit exceeded — try again later"` |
| Error handling: timeout | [PASS] | Line 153: `err "Connection timed out — check network"` |
| Error handling: jq parse | [PASS] | Line 182: `err "Failed to parse API response — check network or HF status"` |
| Error handling: no token | [PASS] | Line 121: `warn "No HF_TOKEN set — using anonymous access"` — continues for public repos per spec |
| All file paths seam-guarded | [PASS] | `CONFIG_FILE="${CONFIG_FILE:-$HOME/…}"` (line 21), `HF_TOKEN="${HF_TOKEN:-}"` (line 22), `HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/…}"` (line 23), re-guarded at line 117 after `--output` override |
| No `err "msg" 1` pattern | [PASS] | All 19 `err` calls use `err "message"` with no trailing exit code — confirmed by grep |

### Convention Compliance

| Item | Status | Evidence |
|------|--------|----------|
| `bash -n` passes | [UNVERIFIED] | Cannot execute `bash -n` due to sandbox restrictions. Builder claims pass. |
| `make gen && make check` passes | [UNVERIFIED] | Cannot execute make. Builder claims pass. Generated files (AGENT_Context, completions) contain correct entries. |
| `make lint` passes (0 FAIL, 0 WARN) | [UNVERIFIED] | Cannot execute make. Builder claims 0 FAIL, 0 WARN. |
| Tool is executable (chmod 100755) | [UNVERIFIED] | Cannot check permissions. `git ls-files` shows file is tracked. Builder claims chmod 100755. |
| POS.md has `ai hf` row + detail block | [PASS] | POS.md line 58: `bin/pos-ai-hf` in `**File:**` line. Lines 103-112: command table + auth/rate-limit/resume detail block. |
| No INTERACTIVE_CMDS change needed | [PASS] | `bin/pos` line 269: `pos-ai-hf` NOT in `INTERACTIVE_CMDS` string — tool does not read stdin. |
| No changes to `bin/pos-ai` | [PASS] | Grep for "hf" in `bin/pos-ai` returns 0 matches. |
| No changes to `bin/pos` | [PASS] | `pos-ai-hf` not in `INTERACTIVE_CMDS`. No other modifications visible. |
| No changes to `preinstall.sh` | [PASS] | Not in git diff. |
| No changes to `lib/common.sh` | [PASS] | Not in git diff. |
| Generated docs contain ai-hf | [PASS] | AGENT_Context: tree line 66, dispatch line 283, filetable line 613. Completions line 6: `_pos_flags[ai-hf]="--branch --gguf --output"` |
| AGENT_TODO Done entry added | [PASS] | Git diff shows new entry at top of Done section, dated 2026-09-04. |

### Security

| Item | Status | Evidence |
|------|--------|----------|
| Token never printed in output | [PASS] | `$HF_TOKEN` referenced only at lines 22, 120, 129-130, 138, 257 — none in any `printf`/`echo` output path. Token warning (line 121) only prints the literal string "No HF_TOKEN set". |
| Token passed via header, not URL | [PASS] | Lines 128-132: `hf_auth_header()` constructs `Authorization: Bearer …` header. Lines 144-146, 260-262: added as `-H` arg to curl. Never in URL string. |
| Config file permissions (ai.env chmod 600) | [UNVERIFIED] | Tool reads from `~/.config/linux_post_install/ai.env`. The chmod is set by `postinstall.sh` (not by this tool). The tool does NOT change permissions — correct behavior. |
| No command injection via repo-id | [PASS] | `repo_id` is user input. Used in: API endpoint construction (line 198-199, passed as curl URL arg — safe), `hf_repo_dir()` (line 214: string substitution `${repo_id//\//-}` — safe), jq filter argument (line 322: `--arg fn "$filename"` — safe), `.hf-meta` heredoc (line 375-382: unquoted heredoc — variables expanded but context is JSON file, not shell execution). No `eval`, no `exec` with user-controlled path. |
| No command injection via filenames | [PASS] | Filenames from API response are parsed by `jq -r` and used in path construction (`$target="${target_dir}/${fname}"`). Passed to `mkdir -p` and curl `-o` — no shell interpretation of the filename value itself. |

---

## Step 1: Subshell Variable Loss in Multi-File Downloads

[FAIL — BLOCKING]

**Finding:** In `cmd_download`, the multi-file download loop at lines 345-366 runs inside a pipe (`printf … | jq … | while IFS= read -r file_json; do … done`). In bash, a pipe creates a subshell, so variables modified inside the `while` loop — specifically `$total_size` (line 349) and `$downloaded` (line 355) — are lost when the pipe exits. The summary section at line 396 reads `$total_size` which is still `0` from its initialization at line 340.

**Severity:** BLOCKING
**Certainty:** FACT — provable from bash subshell semantics. `cmd | while read; do var=...; done` runs the while body in a subshell. Variables set inside do not propagate back.
**Evidence:**
- Line 340: `local total_size=0`
- Line 345: `printf … | jq … | while IFS= read -r file_json; do` — pipe creates subshell
- Line 349: `total_size=$((total_size + fsize))` — modified inside subshell, lost
- Line 396: `total_human="$(hf_human_size "$total_size")"` — reads `0`
**Relevant files/lines:** `bin/pos-ai-hf:340-398`
**Approved scope reference:** Architect Decision 6 (Output Contract) specifies multi-file summary as "📥 Downloaded: meta-llama/Llama-3.1-8B-Instruct (7 files, 4.7 GB)" — the size should be the correct total.
**Why it matters:** Every multi-file download (the common case for large models) will print "0 B" as the total size. This is a user-visible incorrect output and directly violates the Architect's output contract.
**Test mask:** The test harness (Test 22, line 151) checks for `[0-9] B)` which matches "0 B)" — the test passes on the bug. The test needs to check the actual expected sum (618 + 8500000000 + 9000000 + 5000 + 200 + 500 = 8509017318 bytes ≈ "8.5 GB").
**Suggested fix:** Replace the pipe with process substitution (`while IFS= read -r file_json; do … done < <(printf '%s' "$filtered_files" | jq -c '.[]')`) to keep the loop in the main shell, or accumulate total_size via a temp file or another jq pass on `$filtered_files` before the loop.

---

## Step 2: Missing Disk Space Pre-Flight Check

[FAIL — REQUIRED]

**Finding:** The architecture (Decision 5, "Key implementation details" table, and Decision 7, "Error matrix" row) specifies a pre-flight disk space check: `df` available space vs estimated total (from `/tree/` endpoint), with a warning when space is critically low. The implementation has no disk space check at all.

**Severity:** REQUIRED
**Certainty:** FACT — no `df` or `stat`-based space check anywhere in the file (confirmed by grep).
**Relevant files/lines:** `bin/pos-ai-hf` — absent between line 333 (file count check) and line 337 (mkdir).
**Approved scope reference:** Architect Decision 5: "Disk space | Pre-flight check: `df` available space vs estimated total (from `/tree/` endpoint)" and Decision 7: "Disk space | `df` pre-flight | `warn "Low disk space: need {N} GB, only {M} GB available"` then continue (user's call)".
**Why it matters:** Downloading a 7-8 GB model on a near-full disk is a waste of time and leaves partial files. The architecture explicitly chose a non-blocking warning (not an error) — the user makes the final call. Without this, users discover the problem only after curl fails mid-file.
**Suggested fix:** Before the download loop, sum the sizes from `$filtered_files` (this also solves the subshell bug if using jq for the sum), compare with `df --output=avail "$target_dir"`, and `warn` if insufficient. This is a ~5 line addition.

---

## Step 3: Missing `hf_repo_files` API Fallback

[FAIL — REQUIRED]

**Finding:** The architecture specifies `hf_repo_files()` should call `/api/models/{ns}/{repo}/tree/{branch}/` for file sizes, and fall back to `/api/models/{ns}/{repo}` for file list if the tree endpoint fails. The implementation only calls the tree endpoint (line 198) with no fallback.

**Severity:** REQUIRED
**Certainty:** FACT — line 198: `local endpoint="/models/${ns}/${repo}/tree/${branch}"` followed by a single `hf_api "$endpoint"` call. No fallback logic.
**Relevant files/lines:** `bin/pos-ai-hf:188-200`
**Approved scope reference:** Architect Decision 5 (`hf_repo_files()` signature): "Calls: GET /api/models/{ns}/{repo}/tree/{branch}/ for sizes, **falls back to /api/models/{ns}/{repo} for file list**"
**Why it matters:** Some HF repositories (e.g., datasets, some model repos) may not respond to the `/tree/` endpoint (404 or empty). The fallback to `/api/models/{ns}/{repo}` provides a file list (without sizes) so the user can still download. Without it, those repos fail entirely with a 404 error.
**Suggested fix:** Wrap the tree call in a conditional; on 404, call `/api/models/{ns}/{repo}` and construct a minimal `[{"rfilename": "<name>", "size": 0}]` array from the `siblings` array. `size` being 0 is acceptable (displays as "0 B") since the primary goal is getting the download list.

---

## Step 4: Rate-Limit Extra HEAD Request

[NOTE — SUGGESTED]

**Finding:** Line 158 makes a second `curl -sI` HEAD request specifically to extract the `Retry-After` header value after receiving a 429. This is an extra HTTP call that could itself be rate-limited, and the `Retry-After` header was already present in the original request's response (line 151 uses `-w '%{http_code}'` but does not capture response headers).

**Severity:** SUGGESTED
**Certainty:** FACT — line 158: `retry_after="$(curl -sI -H "${auth_header:-}" "$url" 2>/dev/null | grep -i 'retry-after:' | tr -d '\r' | awk '{print $2}')"`
**Relevant files/lines:** `bin/pos-ai-hf:156-161`
**Approved scope reference:** Architect Decision 5: "Rate limiting | Sleep 1s between files; on 429, wait `Retry-After` header value or 60s default"
**Why it matters:** Minor inefficiency. When already rate-limited, making another request is suboptimal. Could use `curl -sS -D -` (dump headers to stdout) in the original request to capture `Retry-After` directly, or simply default to 60s without the extra call.
**Suggested fix:** Change the original curl in `hf_api()` to use `-D -` (or a header dump file) so the `Retry-After` header is available from the first response without a second call.

---

## Step 5: Error Message Format Deviation

[NOTE — SUGGESTED]

**Finding:** The error message for invalid repo format at line 195 (`hf_repo_files`) says `"Invalid repo format: use namespace/model-name"` while the Architect specified `"Invalid repo format: use namespace/model-name"` at Decision 7. However, line 308 (`cmd_download`) also says the same message. The Architect's error matrix entry says `err "Invalid repo format: use namespace/model-name"` — which matches. This is consistent.

However, the Architect's error matrix says the model-not-found message should reference the full `repo-id` (e.g., `err "Model not found: {repo-id}"`), while the implementation at line 175 constructs the message from the API endpoint: `err "Model not found: ${endpoint#/api/models/}"`. The stripped endpoint value is the same as repo-id when the endpoint is `/models/{ns}/{repo}`, but if the endpoint is `/models/{ns}/{repo}/tree/{branch}`, the stripped value would be `{ns}/{repo}/tree/{branch}` — which is confusing.

**Severity:** SUGGESTED
**Certainty:** HYPOTHESIS — only manifests when 404 is returned from `/tree/{branch}` endpoint (which strips to `{ns}/{repo}/tree/{branch}` in the message). The normal `/models/{ns}/{repo}` path produces the correct repo-id in the message.
**Relevant files/lines:** `bin/pos-ai-hf:175`
**Why it matters:** Minor UX: a 404 from the tree endpoint would show a confusing path in the error message instead of the clean repo-id. The Architect's spec just says `{repo-id}`.
**Suggested fix:** Capture the `repo_id` and pass it to `hf_api` or handle the error at the caller level where `repo_id` is available. Or, in `hf_api`, accept an optional display-name parameter for error messages.

---

## Architect Compliance

| Decision | Implemented? | Notes |
|----------|-------------|-------|
| Decision 1: File location `bin/pos-ai-hf` | [YES] | Created at correct path |
| Decision 2: Subcommands (download, search, list, remove) | [YES] | All 4 implemented |
| Decision 3: Config scope extends `ai` | [YES] | `# POS_CONFIG: ai \| ai.env` — correct |
| Decision 4: Download directory layout | [YES] | `<namespace>-<model-name>/` under XDG data dir, `.hf-meta` metadata |
| Decision 5: Download logic | [PARTIAL] | Download flow correct; missing disk space check; missing API fallback |
| Decision 6: Output contract | [PARTIAL] | Emojis, paths correct; multi-file size is always 0 (subshell bug) |
| Decision 7: Error handling | [PARTIAL] | All error matrix cases handled; missing disk space pre-flight |
| Decision 8: Deps/lint compliance | [YES] | POS headers correct, deps before help, source chain |
| Decision 9: Testing strategy | [YES] | Stub-PATH harness exists at expected location with 46 tests |

**Deviations from Architect design:**
1. Disk space pre-flight check not implemented (architect-specified, REQUIRED).
2. `hf_repo_files` API fallback not implemented (architect-specified, REQUIRED).
3. Rate-limit handling uses extra HEAD request instead of extracting from original response (architect did not specify implementation detail — minor deviation).
4. Config loader is an inline pattern rather than copying from `bin/pos-ai` (functionally equivalent, not a deviation in behavior).

---

## Verification Verified

| Claim | Evidence | Status |
|-------|----------|--------|
| Builder: "46/46 tests green" | Test harness exists at `/tmp/opencode/hf-test/run-tests.sh` with 45 numbered tests visible (tests 1-45). Could not execute to confirm count. | UNVERIFIED |
| Builder: "make gen && make check OK" | Generated files (AGENT_Context line 66/283/613, completions line 6) contain correct ai-hf entries. | STRONG INFERENCE |
| Builder: "make lint 0 FAIL, 0 WARN" | All conventions verified by static analysis (POS headers, deps guards, source chain, exec bits claimed). | UNVERIFIED |
| Builder: "bash -n OK" | Cannot execute. No syntax errors visible by manual inspection. | UNVERIFIED |
| Builder: "POS.md updated" | Git diff confirms: ai File line updated (line 58), command table + detail block added (lines 103-112). | FACT |
| Builder: "No changes to bin/pos, bin/pos-ai, preinstall.sh, lib/common.sh" | `git ls-files` confirms these files tracked; grep confirms no hf-related changes. | FACT |

---

## Verification Unverified

| Claim | Reason |
|-------|--------|
| `bash -n` passes | Sandbox prevents execution |
| `make gen && make check` passes | Sandbox prevents execution |
| `make lint` 0 FAIL 0 WARN | Sandbox prevents execution |
| Tool is executable (100755) | Sandbox prevents `ls -la` |
| Test harness 46/46 passes | Sandbox prevents execution |
| Config file ai.env chmod 600 | Controlled by postinstall.sh, not this tool |

---

## Scope Compliance

**In-scope (confirmed):**
- `bin/pos-ai-hf` created with all 4 subcommands
- `DOC/POS.md` ai hf row + detail block
- Generated docs/completions updated via `make gen`
- `AGENT_TODO.md` Done entry

**Out-of-scope (confirmed NOT present):**
- No ollama integration
- No parallel downloads
- No changes to `bin/pos-ai`, `bin/pos`, `preinstall.sh`, `lib/common.sh`
- No new config scope (extends `ai`)
- No `INTERACTIVE_CMDS` change

**Unexpected changes:** None detected.

---

## Remaining Uncertainty

1. Whether `bash -n`, `make gen`, `make check`, and `make lint` actually pass — Builder claims they do, and generated artifacts are consistent with this, but execution was blocked.
2. Whether the 46 test cases actually all pass — the test harness exists with the correct structure, but Test 22 masks the subshell bug (checks for `[0-9] B)` which matches "0 B)").
3. Whether the tool is truly `chmod 100755` — Builder claims it is, file is in git index.
4. Live network behavior against real `huggingface.co` — only stub-tested, not end-to-end verified.

---

## Recommended Next Agent

**Builder**

**Reason:** The 1 BLOCKING finding (subshell variable loss) and 2 REQUIRED findings (disk space check, API fallback) are all within the approved scope and well-understood fixes. The Builder should:
1. Fix the subshell bug (replace pipe with process substitution or pre-compute sum via jq)
2. Add the disk space pre-flight check (~5 lines, `df` + `warn`)
3. Add the `hf_repo_files` fallback to `/api/models/{ns}/{repo}` on 404
4. Update Test 22 to check actual expected size value instead of regex `[0-9] B)`
5. Re-run `bash -n`, tests, and gates

---

## Changes Made by Reviewer

None — read-only review.