Linux_post_install/AgentsReport/explorer/2026-09-06_ai-audit.md

AI Subsystem Audit — Evidence-Based Verification

Date: 2026-09-06 Explorer: read-only, evidence-first. Scope: bin/pos-ai, bin/pos-ai-hf, bin/pos-ai-server, forwarders (gemini/openrouter/llamacpp), lib/ai-providers/*.sh, ai.env config + # POS_CONFIG:/pos config ai, # POS_FLAGS:/# POS_SUBCMDS: metadata, DOC/POS.md ai section, GEN doc rows, completions/pos.bash, bin/pos usage().

---

TL;DR

• The AI subsystem is largely real and coherent. Providers (gemini/openrouter/llamacpp) adapters implement real API calls with key masking; pos-ai-server was genuinely repaired (commit 528b166) — the old fake validate_server_features stub and unquoted multi-line ExecStart are gone, systemd_quote quoting is correct, detect_llama_version is guarded.
• One HIGH defect: pos-ai-hf single-file download failure still exits 0 and writes .hf-meta advertising a complete model (only the parallel/multi-file path was fixed for partial-failure rc=1). bin/pos-ai-hf:716-720, 727-744, 773-775.
• One MEDIUM coherence defect: LLAMACPP_HOST is honored by the server (ExecStart --host + display) but ignored by the llamacpp provider adapter and by the server's own probes, all of which hardcode 127.0.0.1 (lib/ai-providers/llamacpp.sh:14,20,31,46-47; bin/pos-ai-server:131,576).
• pos-ai-server flag validation is NOT complete: only CLI-explicit flags are validated; config-sourced flags and always-emitted defaults (--threads, --n-gpu-layers, --ctx-size, --port, --host) are never validated. "Version-aware" is cosmetic (version only interpolated into the error string, never used to branch logic).
• Streaming is NOT implemented in any provider (llamacpp explicitly sets stream:false); no claim of streaming exists in docs. Not a defect, just a fact to record.
• Command-extraction (_extract_commands / _prompt_run_command) provenance: trusted=1 comes ONLY from the --trust CLI flag or alias wrappers that inject --trust (alias env field 5=1). Confirmation default on a tty is ALLOW (Enter runs); non-tty is fail-safe (never runs). Extra alarm is warranted only if the operator marks an alias trusted.

---

Evidence table

ID	file:line	current behavior	classification	notes
A1	bin/pos-ai:649-677	Parses exactly --provider --model --session --system --full --last --trust + -h	MATCH	# POS_FLAGS: (line 5) matches parse exactly
A2	bin/pos-ai:693-706	Subcommands ask capture chat models providers sessions + llamacpp shorthand	MATCH for the six; llamacpp shorthand IMPLEMENTED-BUT-UNLISTED	# POS_SUBCMDS: (line 4) omits llamacpp; completions/pos.bash:51 does include it
A3	bin/pos-ai:130-160	load_config reads ai.env (env-already-exported wins) + legacy gemini/openrouter files	MATCH	duplicated loader, no shared one (see M3)
A4	bin/pos-ai:162-202	Config precedence: --model > AI_MODEL > provider-specific (AI_GEMINI_MODEL/OPENROUTER_MODEL/LLAMACPP_MODEL) > provider_default_model	MATCH	llamacpp resolve_model passes basename of LLAMACPP_MODEL (line 198)
A5	bin/pos-ai:363-406, 531-535, 571-572	_extract_commands parses bash/sh/shell fenced blocks; _prompt_run_command prompts/executes	MATCH	provenance of trusted traced to --trust flag + alias field 5 (pos-ai-alias:64,433-436,580). tty default = run; non-tty = never runs
A6	lib/ai-providers/gemini.sh:24-36	Real generateContent call; -m 60 timeout; parses error.message, non-200 → rc 1	MATCH	no streaming tokens (?alt=sse absent)
A7	lib/ai-providers/openrouter.sh:24-37	Real /api/v1/chat/completions; -m 60; error.message parse; rc 1 on non-200	MATCH
A8	lib/ai-providers/llamacpp.sh:19-42	Real /v1/chat/completions; -m 120; hardcodes 127.0.0.1, only reads LLAMACPP_PORT; stream:false	MATCH for localhost mode; ignores LLAMACPP_HOST → coherence defect D2	error path only prints "API error $code", no error.message
A9	bin/pos-ai-server:388-392, 445, 501-508	ExecStart single-line, binary+model systemd_quote()d	MATCH	repair confirmed real
A10	bin/pos-ai-server:52-59	detect_llama_version guarded; returns "unknown" safely; takes binary arg	MATCH	repair confirmed; no errexit
A11	bin/pos-ai-server:66-87, 407-409	validate_requested_flags only validates REQUESTED_FLAGS (CLI-explicit only)	PARTIAL (see D3)	config + defaults never validated; grep -qF substring (see D4)
A12	bin/pos-ai-server:262-266, 373-380	Defaults: PORT 8088, HOST 127.0.0.1, CTX 4096, GPU -1, THREADS nproc; CLI > env > default	MATCH	bind localhost by default ✓
A13	bin/pos-ai-server:128-135, 576	health probe /health + /v1/models hardcoded 127.0.0.1	PARTIAL (see D2)	ignores LLAMACPP_HOST for probes
A14	bin/pos-ai-hf:185-251	hf_api auth header, 401/403/404/429/other → err; JSON validity check; 429 retry-once	MATCH	base URL https://huggingface.co/api
A15	bin/pos-ai-hf:254-312	hf_paginate Link rel="next"; hf_repo_files tree endpoint with guards + fallback; hf_search jq urlencode	MATCH	pagination real; error-object guard via select(type=="...")
A16	bin/pos-ai-hf:636-700	Parallel download drain, per-pid wait, honest failure counting, rc=1 on partial failure, no .hf-meta on partial	MATCH for multi-file	single-file path defect = D1
A17	bin/pos-ai-hf:778-812, 915-992	list/cache only list dirs with .hf-meta; cache clear fail-closed confirm	MATCH
A18	bin/pos-ai-hf:449-469	--branch/--revision alias, last-wins; default branch from API else "main"	MATCH
A19	completions/pos.bash:6,7,26,51	completion flags for ai/ai-server/ai-hf match # POS_FLAGS:; subcmds match declared	MATCH for declared	no _pos_subcmds[ai-hf] (see M1)
A20	DOC/POS.md:56-125	ai/hf/server sections describe behavior congruent with code	MATCH	no streaming claim
A21	pos config ai / cfg_display masks secret keys	AI_GEMINI_API_KEY, OPENROUTER_API_KEY, HF_TOKEN are secret-flagged → masked	MATCH	lib/config-ui.sh:347-361

---

Findings (ranked defects)

AI-subystem defects

• D1 (HIGH) — pos-ai-hf single-file download failure returns 0 and writes .hf-meta for a partial model. Evidence: bin/pos-ai-hf:716-720 — sequential path warn "Failed to download $fname"; continue without recording into failed_files; :727-744 — failed_files empty ⇒ .hf-meta written; :773-775 — the return 1 clause only fires for the parallel path. Only if file_count>1 (parallel branch, :641) is a failure non-zero + meta-suppressed. Contradicts commit 528b166 message "rc=1 on partial failure, no .hf-meta for half-downloaded models". A truncated -C - partial .gguf can then be handed to pos ai server start.

• D2 (MEDIUM) — LLAMACPP_HOST advertised but only honored by the server, not by the client/probes. Evidence: server emits --host $HOST (bin/pos-ai-server:445) and displays endpoint: http://$HOST:$PORT (:604), but health probe :131 and /v1/models probe :576 hardcode 127.0.0.1; the llamacpp adapter (lib/ai-providers/llamacpp.sh:14,20,31,46-47) hardcodes 127.0.0.1 and reads only LLAMACPP_PORT. A non-localhost LLAMACPP_HOST → server binds elsewhere while pos ai llamacpp ask/chat/models and probes talk to loopback. Coherence break across the two tools + the advertised config key.

• D3 (MEDIUM) — validate_requested_flags validates only CLI-explicit flags. Evidence: REQUESTED_FLAGS is populated only in the arg-parsing case clauses (bin/pos-ai-server:295-359); the gate if [ "${#REQUESTED_FLAGS[@]}" -gt 0 ] (:407-409) + always-serialized defaults --n-gpu-layers $gpu_layers --ctx-size $CTX_SIZE --threads $THREADS (:446-448) and config-sourced --port/--host (:445) are never checked. Also: the version argument is used only in the error string (:84); there is no version-conditional logic — the "version-aware" phrasing is cosmetic. Practically mitigated because the always-on core flags (--threads, --n-gpu-layers, --ctx-size, --port, --host) are universal across llama.cpp builds, but the guarantee is broader than the implementation.

• D4 (LOW) — validate_requested_flags uses loose substring matching grep -qF -- "$flag" (bin/pos-ai-server:83). A flag token could match inside synthesized --help text unrelated to actual support (e.g. a builder that documents a placeholder), giving false-passes; no word/--flag= boundary. Minor robustness issue.

• D5 (LOW / UNKNOWN-live) — llamacpp model id: resolve_model passes basename "$LLAMACPP_MODEL" (bin/pos-ai:198) as the model field. llama.cpp /v1/models .id is not guaranteed to equal the GGUF basename (may be the model's internal name or full path). If they differ the server may reply "model not found". UNKNOWN — requires live llama-server to confirm.

• D6 (LOW, security-relevant) — command-execution confirmation default is ALLOW on a tty: _prompt_run_command prompt [Y/n] runs on any value except n|N (bin/pos-ai:391-405); trusted=1 (--trust flag :667-668, or alias trusted field via pos-ai-alias:64,433-436) runs with no confirmation (:385-389) via run eval "$cmd" (:388,403). Non-tty is fail-safe (:382 — returns without running). Not an auto-exec bug under default settings, but the confirmation default-allow + eval of AI-extracted shell is worth a security review's attention. Provenance of trusted=1 (question 11): (1) --trust CLI flag, default 0 (:644); (2) alias wrapper inserts --trust when the alias's 5th env field (name|provider|session|prompt|trusted) = 1 (pos-ai-alias:64), set interactively ("Trust this alias? (y/N)", default N) at create/edit (:433-436, 580-591). No config key drives trust. Auto-execution paths: only trusted-mode _prompt_run_command (:388) — nothing else eval's model output.

Metadata / doc drift

• M1 (MEDIUM, metadata gap) — pos-ai-hf implements subcommands search/download/list/remove/info/files/cache but declares no # POS_SUBCMDS: header (bin/pos-ai-hf:3-4). Consequently _pos_subcmds[ai-hf] is absent from completions/pos.bash and the gen doc tables/tree do not surface them. IMPLEMENTED-BUT-UNLISTED.

• M2 (LOW) — pos-ai llamacpp shorthand subcommand (parsed, bin/pos-ai:701-704) absent from # POS_SUBCMDS: (line 4). Completions already include it (completions/pos.bash:51). IMPLEMENTED-BUT-UNLISTED (only in the shorthand).

• M3 (LOW, maintainability) — no shared ai.env loader. pos-ai (load_config, :130), pos-ai-server (load_config, :21), pos-ai-hf (load_hf_config, :30) each duplicate the env-precedence loop; lib/common.sh only provides load_system_env (system.env) and CONFIG_DIR. All three behave identically, but this is a future-drift seam and contradicts the "shared loader" intent implied by AGENTS.md.

• M4 (LOW) — forwarder # POS_SUBCMDS: under-list: pos-ai-openrouter:4 = ask chat sessions capture (omits models, which works via passthrough); all three forwarders omit providers (pos-ai-gemini:4, pos-ai-openrouter:4, pos-ai-llamacpp:4), which also works. Completions inherit the under-list.

• M5 (INFO, not a defect) — QUANT_DIR is a CLI-flag-only variable (bin/pos-ai-hf:119,134-135), not an env/config key; it is not in # POS_CONFIG: and is correctly absent from pos config ai. The audit task listed it among config keys to trace; it is flag-only by design. Configure HF_TOKEN/HF_DOWNLOAD_DIR instead.

Could NOT verify live (read-only / no runtime)

• llama-server binary behavior: --version/--help output, whether --threads/--n-gpu-layers/--gpu-threads/--kv-cache etc. are all accepted by the installed build; the /v1/models id format (D5).
• Actual Hugging Face API responses (tree endpoint shape, pagination Link header, 429 Retry-After); hf_paginate/hf_repo_files correctness with live data.
• systemd unit validity: the reviewer report for 528b166 states systemd-analyze verify rc=0 was checked; I could not re-run it. Static reading of systemd_quote + single-line ExecStart is correct.
• Whether a real Gemini/OpenRouter key/reply path works end-to-end (key required).

---

Handoff

Status: COMPLETE (objective answered to the limit of read-only evidence). Evidence: all file:line cited above; primary source + git commit 528b166 diff.

• Recommended next agent: Reviewer (adversarial review) on the single-file download path (D1) and the LLAMACPP_HOST coherence break (D2) — both are concrete, source-provable behavior defects ripe for a focused fix. Builder if a fix is decided (D1: route single-file through the failure-counting harness or check hf_download_with_progress rc and suppress meta + return 1; D2: thread LLAMACPP_HOST into the adapter and probes). Maintainer for the metadata gaps (M1, M2, M4) and the duplicated ai.env loader (M3).
• Affected areas: bin/pos-ai-hf, bin/pos-ai-server, lib/ai-providers/llamacpp.sh, bin/pos-ai* POS_* headers, completions/pos.bash, gen doc output.
• Decision boundary: adding # POS_SUBCMDS: to pos-ai-hf is a metadata change requiring make gen + make check + make lint (per AGENTS.md) — that is a Builder/Maintainer action, out of Explorer scope.

Remaining uncertainty: D5 (llamacpp model-id match) and all binary/API live behavior (listed above) — needs a live llama.cpp/HF/Gemini environment.