Linux_post_install/AgentsReport/detective/2026-09-04_pos-ai-hf-gguf-jq-bug.md

Detective Report — pos-ai-hf GGUF/jq bug (2026-09-04)

TL;DR

• Status: ROOT_CAUSE_ESTABLISHED
• Symptom: pos ai hf download Qwen/Qwen2.5-3B-Instruct-GGUF --gguf --output ~/.models crashes with jq: error (at <stdin>:0): endswith() requires string inputs (jq exit 5, after [!] No HF_TOKEN set warning).
• Root cause (FACT): hf_repo_files() primary path (bin/pos-ai-hf:199-204) returns the raw HF tree API response, whose entries have keys oid, path, size, type — no rfilename. Every consumer of files_json reads .rfilename → gets null. Line 336 (endswith(".gguf") on null) is the crash site. The user's null-guard alone is insufficient: with the guard, --gguf would silently filter everything out → err "No files to download" (exit 1) instead of downloading the 10 GGUF files. All other modes are also silently broken for every tree-served repo: single-file mode matches nothing, all-files mode writes null into the download URL (404), metadata, and summary.
• Fix: normalize the tree response in hf_repo_files() to [.[] | select(.type == "file") | {rfilename: .path, size: (.size // 0)}] (same shape the fallback already emits), plus a defense-in-depth string guard on the --gguf filter and mode-aware empty-result messages. Verified: 13 files normalize, --gguf selects the 10 .gguf files, single-file/all-files/meta/summary all work unchanged. Live API validation passed (10/10, 0 non-gguf, 0 nulls).
• Expected net change: raw tree result transformed upstream; no semantics change for already-working repos.
• Artifacts: this report; fixtures/harness spec → /tmp/opencode/hf-test2/ (Builder builds it; harness spec in Step 5). No changes made by Detective (read-only).

Step 1: Confirm + quantify the crash and every .rfilename read — [DONE]

Fixture /tmp/opencode/qwen-tree.json (live capture of GET /api/models/Qwen/Qwen2.5-3B-Instruct-GGUF/tree/main): 13 entries, all type:"file", keys per entry oid, path, size, type, no rfilename; 10 entries carry lfs. .path column: .gitattributes, LICENSE, README.md, qwen2.5-3b-instruct-{fp16-00001-of-00002,fp16-00002-of-00002,q2_k,q3_k_m,q4_0,q4_k_m,q5_0,q5_k_m,q6_k,q8_0}.gguf.

Exact reproduction (the exact code, same exit code as the tool — jq exit 5):

$ jq -c '[.[] | select(.rfilename | endswith(".gguf"))]' /tmp/opencode/qwen-tree.json
jq: error (at qwen-tree.json:0): endswith() requires string inputs   # exit=5

Every downstream .rfilename read, observed (not inferred):

Line	Code	Observed with qwen tree	Verdict
333	single-file select(.rfilename == $fn)	[] for any fn (null == "README.md" → false; exit 0)	silent no-match → err "No files to download"
336	--gguf select(.rfilename | endswith(".gguf"))	jq error exit 5 (the reported crash)	the crash
339	all-files jq -c '.'	passes all 13 (no filter)	nothing filtered, but downstream 372 breaks
372	loop jq -r '.rfilename'	null ×13	URL …/resolve/main/null → 404; curl -o left an empty null file; loop warned
398	meta [.[] | .rfilename]	[null,null,…13]	.hf-meta files list all null
412	one-file summary .[0].rfilename	null	Downloaded: …/null

jq empty (line 181), .size // 0 (lines 373, 414), [.[].size // 0] \| add // 0 (line 353): unaffected — size handling is already null-safe.

Fallback shape verified live (GET /api/models/Qwen/Qwen2.5-3B-Instruct-GGUF → jq '[.siblings[] | {rfilename: .rfilename, size: (.size // 0)}]'): 13 entries, 0 null rfilename, all size: 0 (metadata API has no per-sibling sizes). Shape {rfilename, size} — exactly what the normalization produces for the tree path. Fallback path needs no change.

Step 2: Unsafe-jq sweep (whole file bin/pos-ai-human) — [DONE]

All jq expressions in bin/pos-ai-hf, with assessment (only c/p rfilename-related ones are the bug family):

Line	Expression	Assessment
181	jq empty on API body	validation only; safe
210	[.siblings[]|{rfilename:.rfilename, size:(.size//0)}]	correct shape; null-safe; leave as is
217	jq -sRr @uri (query encode)	safe
254	.defaultBranch // empty	null-safe; safe
298	jq 'length' (search)	safe
302	.[] | "…\(.id)…\(.downloads // 0)…\(.likes // 0)"	search API provides these; // 0 guards; safe
333	select(.rfilename == $fn)	AFFECTED: null vs string → silently []. Fixed by normalization (works after); no other string-op risk.
336	select(.rfilename | endswith(".gguf"))	THE CRASH. Category (a): string function on possibly-null field.
339	jq -c '.'	passthrough; safe
343	jq 'length'	safe
353	[.[].size // 0] | add // 0	null-safe on size; safe
372	jq -r '.rfilename'	Affected: prints literal null → bad URL/404 + empty null target file
373	jq -r '.size // 0'	null-safe; safe
398	jq -c '[.[] | .rfilename]'	Affected: meta list all nulls
412	jq -r '.[0].rfilename'	Affected: summary prints null
414	jq -r '.[0].size // 0'	null-safe; safe
447	jq -r '.downloaded_at // "unknown"' (meta file)	safe; meta file is JSON

Category (a) string-function-on-null type: only line 336 in this file (no startswith/contains/test in pos-ai-hf at all — grep confirmed; the other matches above are in other tools/service files, out of scope). Category (b) assumes-field-primary-API-returns: only the rfilename family above (lines 333/336/372/398/412). Category (c) covered in Step 1. Category (d) silent no-match on null: line 333 (only one). No other crash-class bugs found; the rfilename family is the whole story.

Related-but-out-of-scope notes (observations, not part of this fix):

• hf_api uses curl -sS without -L; HF redirects some aliases (verified: runwayml/stable-diffusion-v1-5/tree/main → 307 → stable-diffusion-v1-5/stable-diffusion-v1-5). Such repos fail on BOTH tree and fallback (API request failed (HTTP 307)). Pre-existing; unrelated to this bug; would need -L or canonical-resolution; flag to Builder/Architect, don't fold in.
• Tree endpoint is non-recursive; repos with subdirectories (e.g. SD-v1-5: feature_extractor/, …) only list top-level entries + type:"directory" markers. Fix filters out directories → all-files mode skips subdir files (same as pre-bug behavior; tree path never listed them). Optional follow-up: ?recursive=true — requires # POS_FLAGS/docs change, NOT part of this minimal fix.
• hf_download_file URL building concatenates raw path into URL; files with spaces would need URL-encoding (@uri). Pre-existing; rare for models; not this bug.
• Names that are -, . etc. unaffected.

3. hf_download_file / hf_api related to THIS bug — [DONE]

• hf_download_file (264-286): no field assumptions of its own; takes URL+target. It is a victim: with raw-tree null rfilename, URL …/resolve/main/null returns 404, curl fails → warn "Download interrupted for null (resume…)", and an empty null file remains in the model dir (then cmd_list counts it as size 0). After normalization the function works as designed (has -L for HF's 302→CDN; -C - resume; empty-file guard). No change needed.
• hf_api (134-186): 200/401/403/404/429 handling + jq empty validation — nothing rfilename-related. No change needed (the 307 note in Step 2 is separate).
• hf_resolve_branch (241-261): live-verified defaultBranch "main" resolves fine; unaffected.

4. Fix spec — Builder-executable — [DONE]

Objective: make the primary tree path emit the same {rfilename, size} shape the rest of the file (and the fallback path) already assume. Minimal, CLI semantics preserved (all options, filters, messages keep their meaning; only multi-mode empty-result messages get mode-specific text).

Where: hf_repo_files() body, primary branch, current lines 198-204.

Change 1 — normalize tree response (the fix). Replace:

    local endpoint="/models/${ns}/${repo}/tree/${branch}"
    local result
    if result="$(hf_api "$endpoint" 2>/dev/null)"; then
        printf '%s' "$result"
        return 0
    fi

with (exact code for Builder):

    local endpoint="/models/${ns}/${repo}/tree/${branch}"
    local result
    if result="$(hf_api "$endpoint" 2>/dev/null)"; then
        # Tree API returns {type,path,size,oid[,lfs]} per entry — normalize to the
        # {rfilename,size} shape the rest of the pipeline expects (same as fallback).
        # Skip "directory" entries: they have no resolvable file URL.
        printf '%s' "$result" | jq '[.[] | select(.type == "file") | {rfilename: .path, size: (.size // 0)}]'
        return 0
    fi

• This reuses the exact jq via one pipe; set -euo pipefail semantics: if the transform ever fails, result was already valid JSON so it fails before the pipe — fine (same failure mode as a jq typo elsewhere).
• .size // 0 covers entries lacking size (dirs carry size 0; all observed files carry real size incl. LFS files, whose top-level size is the true byte size).
• Do NOT change the fallback (lines 210-211) — it already emits {rfilename, size} (had the same shape requirement; verified live).
• Edge: tree returning a non-array (code-200 error object) — [.[] | select… | {…}] yields [{rfilename:null}]-style or []; --gguf guard + Step-3 messages convert that to a graceful error. Acceptable; no extra guard required.

Change 2 — defense-in-depth guard in the --gguf filter (line 336). Recommended, type-check form (strictly safe even if rfilename were a non-string non-null):

        filtered_files="$(printf '%s' "$files_json" | jq -c '[.[] | select((.rfilename | type) == "string" and (.rfilename | endswith(".gguf")))]')"

Equivalent accepted: select(((.rfilename // "") | endswith(".gguf")))} — both are purely defensive here (normalized data always strings); must NOT become a replacement for Change-2-Normalization: with normalization, guard-or-not both select the 10 gguf. Verified equivalent on fixture: guarded 10, unguarded 10.

Change 3 — mode-aware "no files" message (replaces line 344, [ "$file_count" -gt 0 ] || err "No files to download"). Keep exit-1 semantics, distinct messages per mode:

    local file_count
    file_count="$(printf '%s' "$filtered_files" | jq 'length')"
    if [ "$file_count" -eq 0 ]; then
        if [ -n "$filename" ]; then
            err "File not found: $filename in $repo_id (branch: ${branch})"
        elif [ "$GGUF_ONLY" -eq 1 ]; then
            err "No .gguf files found in $repo_id${branch:+ (branch: $branch)} — try without --gguf"
        else
            err "No files to download"
        fi
    fi

Not required for the crash fix; required by edge-case spec (no-gguf repo → graceful, distinct message, not crash), and fixes the misleading "No files to download" in single-file mode.

Line content checks after Changes 1-3 (verified by fixture/live):

• single-file line 333: fits; select(.rfilename == $fn) on normalized → 1 for exact README.md / qwen2.5-3b-instruct-q4_k_m.gguf.
• all-files line 339: fits; loop line 372 pulls real rfilename; size line 373 real; meta line 398 real list; summary 412 real.
• Lines 372/373/376/377/398/412 need NO change once normalized (checked on fixture).

5. Edge cases — [DONE]

Case	Behavior before fix	After fix
Repo w/ only type:"directory" (tree)	null → crash/downstream; e.g. gguf mode crashes, all-files nulls	select(.type=="file") → [] → mode-aware graceful error
Empty array / empty siblings tree	crash / silent	[] → graceful error
--gguf on repo w/o .gguf	crash	err "No .gguf files found in …" (exit 1, no crash)
File entry missing size / size:0	.size // 0 everywhere → ok	unchanged; normalization also // 0
LFS files (.gguf 2GB+)	n/a (never reached)	sizes real (2104932768); disk pre-check works
Repo w/ subdirs (non-recursive tree)	null loop	dirs filtered; subtree files not listed — pre-existing semantics (flag in Step 2, decision boundary for follow-up only)
--branch non-main	field absent regardless	branch is only URL+tree-parameter; normalized same way

6. Test plan (harness spec for Builder) — [DONE]

Location: /tmp/opencode/hf-test2/ (workspace must NOT gain test files; repo has no harness for pos-ai-hf; user requirement = new fixture-based harness).

Key constraint — DO NOT source bin/pos-ai-hf in the harness: the tool executes flag parsing + usage/cmd_download at top level (set -euo pipefail; no-args → usage → exit 0 is a NAK). The harness must test the jq transforms in isolation (option b of the brief). Extraction of the functions via sed -n is not recommended (fragile); fix the documented transforms — the transforms ARE the bug.

Fixture files (create in harness setup, static content):

• fixtures/tree-files.json — copy of /tmp/opencode/qwen-tree.json (13 files, rfilename ABSENT; 10 .gguf).
• fixtures/tree-with-dirs.json — copy of /tmp/opencode/sd-tree.json (8 files + 7 type:"directory").
• fixtures/tree-empty.json — [].
• fixtures/tree-nogguf.json — e.g. [{"type":"file","path":"README.md","size":100}].
• Set at top: NORM='[.[] | select(.type == "file") | {rfilename: .path, size: (.size // 0)}]' and GGUF_FILTER='[.[] | select((.rfilename | type) == "string" and (.rfilename | endswith(".gguf")))]' FN_FILTER='[.[] | select(.rfilename == $fn)]' — duplicated strings; if either diverges from the file, tests catch drift when grep -F checks below run.

Assertions (each a small t_<name> function; count PASS/FAIL; exit non-zero on any fail):

1. t_tree_normalize: jq -c "$NORM" tree-001.json → length 13; every .size is number; no entry has rfilename == null.
2. t_gguf_on_normalized: pipe NORM(tree-001) → GGUF_FILTER → length 10; contains qwen2.5-3b-instruct-q4_k_m.gguf; NOT contains README.md/LICENSE/.gitattributes.
3. t_single_file: jq -c --arg fn "qwen2.5-3b-instruct-q4_k_m.gguf" "$FN_FILTER" on NORM(tree-001) → length 1; on --arg fn "no-such-file" → 0 (no crash).
4. t_all_files_passthrough: jq -c '.' → 13; loop pipe .[] → 13 rows, each with string rfilename (no null).
5. t_empty: NORM on tree-empty.json → [] | run the mode-0 guard → error message path (grep the code); i.e. assert printf '[]' | jq "$NORM" outputs [] and file_count logic (replicate [ "$(…|jq 'length')" -eq 0 ]) succeeds.
6. t_nogguf: NORM(tree-nogguf.json) → GGUF_FILTER → length 0, no crash; assert the code contains the "No .gguf files" branch (grep).
7. t_defense_guard: pipe raw qwen-tree.json (unnormalized) through GGUF_FILTER → length 0, exit 0 (proves guard null-proof and non-weakening: the sole difference 10→0 is caused by normalization, guard itself no-op).
8. t_dirs_excluded: NORM(tree-with-dirs.json) → length 8 (files only; 7 dirs dropped).
9. t_code_sync (drift check): grep -Fq 'select(.type == "file")' <repo>/bin/pos-ai-hf and grep -Fq 'endswith(".gguf")' present — catches D it if Builder changed jq inline, test stays honest.

Live smoke (optional; fast, no download): curl tree for Qwen → NORM → GGUF_FILTER → assert 10 rfilenames, 0 README. (This exact pipe was executed in Step 1; pass.)

7. Verification commands for Builder (after implementing)

• bash -n bin/pos-ai-hf (repo copy — the installed /usr/local/bin copy is byte-identical; contract must be fixed in the repo copy).
• bash /tmp/opencode/hf-test2/run-tests.sh → all PASS.
• make gen && make check && make lint → doc tables/registry unchanged; expect green, 0 FAIL, 0 WARN, and git diff limited to bin/pos-ai-hf (+ any doc touch required by CONVENTION pointers; no GEN:START/END blocks change).
• Live no-download validation (already demonstrated passing):

curl -sS "https://huggingface.co/api/models/Qwen/Qwen2.5-3B-Instruct-GGUF/tree/main" \
  | jq '[.[] | select(.type=="file") | {rfilename:.path,size:(.size//0)}]' \
  | jq '[.[] | select(.rfilename|endswith(".gguf"))]' | jq 'length'   # expect 10, no README/LICENSE

• Optional tiny-download proof: pos ai hf download Qwen/Qwen2.5-3B-Instruct-GGUF LICENSE --output /tmp/hf-small → expect .hf-meta listing LICENSE and 1-file summary with real size; deletes nothing else.
• Full ~2GB download verified: OUT OF SCOPE (explicit).

Handoff

Status: ROOT_CAUSE_ESTABLISHED

• Symptom: crash jq: endswith() requires string inputs (line 336) on --gguf; silent no-match single-file; null URLs/meta/summary in all-files.
• Expected vs actual: primary tree response should look like the siblings metadata (rfilename-keyed) but arrives free-.rfilename keys; first divergence = hf_repo_files returns raw tree (line 201-204).
• Root cause: missing shape normalization of /tree response in hf_repo_files() — user's endswith guard insufficient (guarded --gguf would download 0 files / "No files to download").
• Classification: FACT (crash & downstream effects reproduced on live fixture; normalization + guard live-validated elsewhere).
• Alternatives eliminated: (a) network/API failure — endpoints live 200 & JSON; (b) rfilename present but null — keys are absent (but oid,path,size,type), confirmed on fetch; (c) fallback path defect — live-verified correct shape; (d) curl/URL issue in hf_download_file — reached only if loop got a non-null name; function itself defect-free.
• Affected components: bin/pos-ai-hf — hf_repo_files() (primary branch), cmd_download() lines 333/336/341-344 (message branch), and downstream read sites 372/398/412 (no change needed once normalized).
• Recommended next agent: Builder — fix is exactly scoped: one transform in hf_repo_files(), one defense-in-depth guard line, one message branch; implement per spec Step 4 and run Step 7 verification. Tester (after Builder) — no committed harness; new /tmp/opencode/hf-test2 harness + optional repeat live tests; recommend adding to repo test suite if project adopts (decision boundary: Architect).
• Remaining uncertainty: none material on the bug; only flagged out-of-scope items (hf_api -L/307 for alias repos; non-recursive tree semantics) for maintainers.
• Changes made by Detective: none (read-only).