# AI Subsystem Audit — Evidence-Based Verification

Date: 2026-09-06
Explorer: read-only, evidence-first.
Scope: `bin/pos-ai`, `bin/pos-ai-hf`, `bin/pos-ai-server`, forwarders (`gemini`/`openrouter`/`llamacpp`), `lib/ai-providers/*.sh`, `ai.env` config + `# POS_CONFIG:`/`pos config ai`, `# POS_FLAGS:`/`# POS_SUBCMDS:` metadata, `DOC/POS.md` ai section, GEN doc rows, `completions/pos.bash`, `bin/pos` usage().

---

## TL;DR

- **The AI subsystem is largely real and coherent.** Providers (gemini/openrouter/llamacpp) adapters implement real API calls with key masking; `pos-ai-server` was genuinely repaired (commit `528b166`) — the old fake `validate_server_features` stub and unquoted multi-line ExecStart are gone, `systemd_quote` quoting is correct, `detect_llama_version` is guarded.
- **One HIGH defect:** `pos-ai-hf` **single-file** download failure still exits **0** and writes `.hf-meta` advertising a complete model (only the parallel/multi-file path was fixed for partial-failure rc=1). `bin/pos-ai-hf:716-720, 727-744, 773-775`.
- **One MEDIUM coherence defect:** `LLAMACPP_HOST` is honored by the server (ExecStart `--host` + display) but **ignored** by the llamacpp provider adapter and by the server's own probes, all of which hardcode `127.0.0.1` (`lib/ai-providers/llamacpp.sh:14,20,31,46-47`; `bin/pos-ai-server:131,576`).
- **pos-ai-server flag validation is NOT complete:** only CLI-explicit flags are validated; config-sourced flags and always-emitted defaults (`--threads`, `--n-gpu-layers`, `--ctx-size`, `--port`, `--host`) are never validated. "Version-aware" is cosmetic (version only interpolated into the error string, never used to branch logic).
- **Streaming is NOT implemented** in any provider (llamacpp explicitly sets `stream:false`); **no claim** of streaming exists in docs. Not a defect, just a fact to record.
- **Command-extraction** (`_extract_commands` / `_prompt_run_command`) provenance: `trusted=1` comes ONLY from the `--trust` CLI flag or alias wrappers that inject `--trust` (alias env field 5=1). Confirmation default on a tty is **ALLOW** (Enter runs); non-tty is fail-safe (never runs). Extra alarm is warranted only if the operator marks an alias trusted.

---

## Evidence table

| ID | file:line | current behavior | classification | notes |
|----|-----------|------------------|----------------|-------|
| A1 | bin/pos-ai:649-677 | Parses exactly `--provider --model --session --system --full --last --trust` + `-h` | MATCH | `# POS_FLAGS:` (line 5) matches parse exactly |
| A2 | bin/pos-ai:693-706 | Subcommands `ask capture chat models providers sessions` + `llamacpp` shorthand | MATCH for the six; **llamacpp shorthand IMPLEMENTED-BUT-UNLISTED** | `# POS_SUBCMDS:` (line 4) omits `llamacpp`; completions/pos.bash:51 does include it |
| A3 | bin/pos-ai:130-160 | `load_config` reads ai.env (env-already-exported wins) + legacy gemini/openrouter files | MATCH | duplicated loader, no shared one (see M3) |
| A4 | bin/pos-ai:162-202 | Config precedence: `--model` > AI_MODEL > provider-specific (`AI_GEMINI_MODEL`/`OPENROUTER_MODEL`/`LLAMACPP_MODEL`) > `provider_default_model` | MATCH | llamacpp `resolve_model` passes **basename** of LLAMACPP_MODEL (line 198) |
| A5 | bin/pos-ai:363-406, 531-535, 571-572 | `_extract_commands` parses ```bash/sh/shell``` fenced blocks; `_prompt_run_command` prompts/executes | MATCH | provenance of trusted traced to `--trust` flag + alias field 5 (pos-ai-alias:64,433-436,580). tty default = run; non-tty = never runs |
| A6 | lib/ai-providers/gemini.sh:24-36 | Real `generateContent` call; `-m 60` timeout; parses error.message, non-200 → rc 1 | MATCH | no streaming tokens (`?alt=sse` absent) |
| A7 | lib/ai-providers/openrouter.sh:24-37 | Real `/api/v1/chat/completions`; `-m 60`; error.message parse; rc 1 on non-200 | MATCH | |
| A8 | lib/ai-providers/llamacpp.sh:19-42 | Real `/v1/chat/completions`; `-m 120`; **hardcodes `127.0.0.1`**, only reads `LLAMACPP_PORT`; `stream:false` | MATCH for localhost mode; **ignores LLAMACPP_HOST** → coherence defect D2 | error path only prints "API error $code", no error.message |
| A9 | bin/pos-ai-server:388-392, 445, 501-508 | ExecStart single-line, binary+model `systemd_quote()`d | MATCH | repair confirmed real |
| A10 | bin/pos-ai-server:52-59 | `detect_llama_version` guarded; returns "unknown" safely; takes binary arg | MATCH | repair confirmed; no errexit |
| A11 | bin/pos-ai-server:66-87, 407-409 | `validate_requested_flags` only validates `REQUESTED_FLAGS` (CLI-explicit only) | PARTIAL (see D3) | config + defaults never validated; grep -qF substring (see D4) |
| A12 | bin/pos-ai-server:262-266, 373-380 | Defaults: PORT 8088, HOST 127.0.0.1, CTX 4096, GPU -1, THREADS nproc; CLI > env > default | MATCH | bind localhost by default ✓ |
| A13 | bin/pos-ai-server:128-135, 576 | health probe `/health` + `/v1/models` **hardcoded `127.0.0.1`** | PARTIAL (see D2) | ignores LLAMACPP_HOST for probes |
| A14 | bin/pos-ai-hf:185-251 | `hf_api` auth header, 401/403/404/429/other → `err`; JSON validity check; 429 retry-once | MATCH | base URL https://huggingface.co/api |
| A15 | bin/pos-ai-hf:254-312 | `hf_paginate` Link rel="next"; `hf_repo_files` tree endpoint with guards + fallback; `hf_search` jq urlencode | MATCH | pagination real; error-object guard via `select(type=="...")` |
| A16 | bin/pos-ai-hf:636-700 | Parallel download drain, per-pid wait, honest failure counting, `rc=1` on partial failure, no `.hf-meta` on partial | MATCH for multi-file | **single-file path defect = D1** |
| A17 | bin/pos-ai-hf:778-812, 915-992 | `list`/`cache` only list dirs with `.hf-meta`; `cache clear` fail-closed confirm | MATCH | |
| A18 | bin/pos-ai-hf:449-469 | `--branch`/`--revision` alias, last-wins; default branch from API else "main" | MATCH | |
| A19 | completions/pos.bash:6,7,26,51 | completion flags for ai/ai-server/ai-hf match `# POS_FLAGS:`; subcmds match declared | MATCH for declared | **no `_pos_subcmds[ai-hf]`** (see M1) |
| A20 | DOC/POS.md:56-125 | ai/hf/server sections describe behavior congruent with code | MATCH | no streaming claim |
| A21 | pos config ai / cfg_display masks `secret` keys | AI_GEMINI_API_KEY, OPENROUTER_API_KEY, HF_TOKEN are `secret`-flagged → masked | MATCH | lib/config-ui.sh:347-361 |

---

## Findings (ranked defects)

### AI-subystem defects

- **D1 (HIGH)** — `pos-ai-hf` single-file download failure returns **0** and writes `.hf-meta` for a partial model.
  Evidence: `bin/pos-ai-hf:716-720` — sequential path `warn "Failed to download $fname"; continue` without recording into `failed_files`; `:727-744` — `failed_files` empty ⇒ `.hf-meta` written; `:773-775` — the `return 1` clause only fires for the parallel path. Only if `file_count>1` (parallel branch, `:641`) is a failure non-zero + meta-suppressed. Contradicts commit `528b166` message "rc=1 on partial failure, no .hf-meta for half-downloaded models". A truncated `-C -` partial `.gguf` can then be handed to `pos ai server start`.

- **D2 (MEDIUM)** — `LLAMACPP_HOST` advertised but only honored by the server, not by the client/probes.
  Evidence: server emits `--host $HOST` (`bin/pos-ai-server:445`) and displays `endpoint: http://$HOST:$PORT` (`:604`), but health probe `:131` and `/v1/models` probe `:576` hardcode `127.0.0.1`; the llamacpp **adapter** (`lib/ai-providers/llamacpp.sh:14,20,31,46-47`) hardcodes `127.0.0.1` and reads only `LLAMACPP_PORT`. A non-localhost `LLAMACPP_HOST` → server binds elsewhere while `pos ai llamacpp ask/chat/models` and probes talk to loopback. Coherence break across the two tools + the advertised config key.

- **D3 (MEDIUM)** — `validate_requested_flags` validates **only CLI-explicit** flags.
  Evidence: `REQUESTED_FLAGS` is populated only in the arg-parsing `case` clauses (`bin/pos-ai-server:295-359`); the gate `if [ "${#REQUESTED_FLAGS[@]}" -gt 0 ]` (`:407-409`) + always-serialized defaults `--n-gpu-layers $gpu_layers --ctx-size $CTX_SIZE --threads $THREADS` (`:446-448`) and config-sourced `--port/--host` (`:445`) are never checked. Also: the `version` argument is used **only** in the error string (`:84`); there is no version-conditional logic — the "version-aware" phrasing is cosmetic. Practically mitigated because the always-on core flags (`--threads`, `--n-gpu-layers`, `--ctx-size`, `--port`, `--host`) are universal across llama.cpp builds, but the guarantee is broader than the implementation.

- **D4 (LOW)** — `validate_requested_flags` uses loose substring matching `grep -qF -- "$flag"` (`bin/pos-ai-server:83`). A flag token could match inside synthesized `--help` text unrelated to actual support (e.g. a builder that documents a placeholder), giving false-passes; no word/`--flag=` boundary. Minor robustness issue.

- **D5 (LOW / UNKNOWN-live)** — llamacpp model id: `resolve_model` passes `basename "$LLAMACPP_MODEL"` (`bin/pos-ai:198`) as the `model` field. llama.cpp `/v1/models` `.id` is not guaranteed to equal the GGUF basename (may be the model's internal name or full path). If they differ the server may reply "model not found". UNKNOWN — requires live llama-server to confirm.

- **D6 (LOW, security-relevant)** — command-execution confirmation default is **ALLOW** on a tty: `_prompt_run_command` prompt `[Y/n]` runs on any value except `n|N` (`bin/pos-ai:391-405`); trusted=1 (`--trust` flag `:667-668`, or alias trusted field via `pos-ai-alias:64,433-436`) runs with **no** confirmation (`:385-389`) via `run eval "$cmd"` (`:388,403`). Non-tty is fail-safe (`:382` — returns without running). Not an auto-exec bug under default settings, but the confirmation default-allow + `eval` of AI-extracted shell is worth a security review's attention.
  *Provenance of trusted=1 (question 11):* (1) `--trust` CLI flag, default 0 (`:644`); (2) alias wrapper inserts `--trust` when the alias's 5th env field (`name|provider|session|prompt|trusted`) = 1 (`pos-ai-alias:64`), set interactively ("Trust this alias? (y/N)", default N) at create/edit (`:433-436, 580-591`). **No config key** drives trust. Auto-execution paths: only trusted-mode `_prompt_run_command` (`:388`) — nothing else eval's model output.

### Metadata / doc drift

- **M1 (MEDIUM, metadata gap)** — `pos-ai-hf` implements subcommands `search/download/list/remove/info/files/cache` but declares **no `# POS_SUBCMDS:` header** (`bin/pos-ai-hf:3-4`). Consequently `_pos_subcmds[ai-hf]` is absent from `completions/pos.bash` and the gen doc tables/tree do not surface them. IMPLEMENTED-BUT-UNLISTED.

- **M2 (LOW)** — `pos-ai` `llamacpp` shorthand subcommand (parsed, `bin/pos-ai:701-704`) absent from `# POS_SUBCMDS:` (line 4). Completions already include it (`completions/pos.bash:51`). IMPLEMENTED-BUT-UNLISTED (only in the shorthand).

- **M3 (LOW, maintainability)** — no shared `ai.env` loader. pos-ai (`load_config`, `:130`), pos-ai-server (`load_config`, `:21`), pos-ai-hf (`load_hf_config`, `:30`) each duplicate the env-precedence loop; `lib/common.sh` only provides `load_system_env` (system.env) and `CONFIG_DIR`. All three behave identically, but this is a future-drift seam and contradicts the "shared loader" intent implied by AGENTS.md.

- **M4 (LOW)** — forwarder `# POS_SUBCMDS:` under-list: `pos-ai-openrouter:4` = `ask chat sessions capture` (omits `models`, which works via passthrough); all three forwarders omit `providers` (`pos-ai-gemini:4`, `pos-ai-openrouter:4`, `pos-ai-llamacpp:4`), which also works. Completions inherit the under-list.

- **M5 (INFO, not a defect)** — `QUANT_DIR` is a CLI-flag-only variable (`bin/pos-ai-hf:119,134-135`), not an env/config key; it is not in `# POS_CONFIG:` and is correctly absent from `pos config ai`. The audit task listed it among config keys to trace; it is flag-only by design. Configure `HF_TOKEN`/`HF_DOWNLOAD_DIR` instead.

### Could NOT verify live (read-only / no runtime)

- llama-server binary behavior: `--version`/`--help` output, whether `--threads`/`--n-gpu-layers`/`--gpu-threads`/`--kv-cache` etc. are all accepted by the installed build; the `/v1/models` id format (D5).
- Actual Hugging Face API responses (tree endpoint shape, pagination Link header, 429 Retry-After); `hf_paginate`/`hf_repo_files` correctness with live data.
- systemd unit validity: the reviewer report for `528b166` states `systemd-analyze verify` rc=0 was checked; I could not re-run it. Static reading of `systemd_quote` + single-line ExecStart is correct.
- Whether a real Gemini/OpenRouter key/reply path works end-to-end (key required).

---

## Handoff

Status: COMPLETE (objective answered to the limit of read-only evidence).
Evidence: all file:line cited above; primary source + git commit `528b166` diff.

- **Recommended next agent: Reviewer (adversarial review)** on the single-file download path (D1) and the `LLAMACPP_HOST` coherence break (D2) — both are concrete, source-provable behavior defects ripe for a focused fix. **Builder** if a fix is decided (D1: route single-file through the failure-counting harness or check `hf_download_with_progress` rc and suppress meta + return 1; D2: thread `LLAMACPP_HOST` into the adapter and probes). **Maintainer** for the metadata gaps (M1, M2, M4) and the duplicated ai.env loader (M3).
- Affected areas: `bin/pos-ai-hf`, `bin/pos-ai-server`, `lib/ai-providers/llamacpp.sh`, `bin/pos-ai*` POS_* headers, `completions/pos.bash`, gen doc output.
- Decision boundary: adding `# POS_SUBCMDS:` to `pos-ai-hf` is a metadata change requiring `make gen` + `make check` + `make lint` (per AGENTS.md) — that is a Builder/Maintainer action, out of Explorer scope.

Remaining uncertainty: D5 (llamacpp model-id match) and all binary/API live behavior (listed above) — needs a live llama.cpp/HF/Gemini environment.