0b5043a9f3
gates / consistency-and-conventions (push) Successful in 26s
User report after the llamacpp app install: 'installed llama.cpp unknown', valid flags rejected (randomly per run), 'Model not found' for the HF downloader's own layout, and a systemd user-bus failure over SSH. Detective (real b10822 binary, FACT) found four independent causes: - version: llama-server --version prints to STDERR; detect_llama_version's 2>/dev/null swallowed it -> always 'unknown'. Now captures 2>&1 + accepts semver/build tokens (incl. build 1.2.3 edge) - validation: printf|grep -q under pipefail -> SIGPIPE rc=141 race randomly rejected flags present in the 59 KB --help. Now pipe-less grep (no race); 20x determinism regression test - model resolution: resolve_model accepted files only, but the HF downloader creates <models>/<repo>/file.gguf dirs. Now expands a dir with exactly one *.gguf (never silently picks; multi-gguf lists + errs) - port: llama.cpp default 8080 vs tool/adapter 8088; validation reliability means --port is now always pinned in the unit - user bus: headless/SSH sessions lack XDG_RUNTIME_DIR -> ensure_user_bus in lib/common.sh pre-flights all three systemctl --user tools with remediation text; pos ai server --no-unit direct-run escape hatch (pidfile) for boxes with no bus - find_llamacpp narrowed to llama-server/llama-server-cuda (bare 'server' fallback hazard); installer post-install sanity (version+help execute, symlink targets resolve) Architect decisions DQ1-DQ6 recorded. Tester: 4 new regression files (version-from-stderr, 25x flag-validation determinism, model dir expansion, bus pre-flight + E2E) + 3 fixture updates; suite 16 files / 269 checks. Verified: make gen idempotent; make check OK; make lint 0 FAIL, 0 WARN; make test 269/269 (~49s); bash -n clean; git diff --check clean.
282 lines
23 KiB
Markdown
282 lines
23 KiB
Markdown
# Detective Report: `pos ai server start` — llama.cpp Post-Install Breakage
|
||
|
||
**Date:** 2026-09-06
|
||
**Investigator:** Detective (read-only)
|
||
**Symptom (user paste):** After `bash apps/install.sh llamacpp` installed llama.cpp b10822, `pos ai server start Qwen-Qwen3-1.7B-GGUF` fails: version "unknown", default flags rejected, model-not-found, and dbus error at the end.
|
||
**Severity:** High — complete server-start failure after a clean install; the generated unit is silently corrupted.
|
||
|
||
## TL;DR
|
||
|
||
| Hyp | Verdict | One-line evidence |
|
||
|-----|---------|-------------------|
|
||
| H1 | **CONFIRMED** (wrong mechanism) | `llama-server --version` prints to **stderr** (`common/build-info.h:13` default `FILE* = stderr`); `detect_llama_version` discards it with `2>/dev/null` → always "unknown" |
|
||
| H2 | **REFUTED as user's cause; latent defect stands** | Archive ships `llama-server`; installer symlinks it (verified); but bare `server` fallback picks unrelated binaries when llama-server is missing (fixture-proven) |
|
||
| H3 | **REFUTED as stated; REAL bug found** | Word-boundary regex MATCHES all 5 flags in real help; the actual failure is `printf|grep -q` + `set -o pipefail` **SIGPIPE race** (pipeline rc=141) — flaky per-run |
|
||
| H4 | **CONFIRMED** | `resolve_model` accepts only files; a dir containing one `.gguf` under `HF_DOWNLOAD_DIR` fails with exact user error (reproduced) |
|
||
| H5 | **CONFIRMED** | Unit written (`:584-602`) BEFORE `systemctl --user daemon-reload` (`:605`); no bus pre-check; SSH without `XDG_RUNTIME_DIR` → exact error, `set -e` aborts, linger hint never runs; orphaned unit remains |
|
||
| H6 | **CONFIRMED** | llama.cpp default port **8080** (`common/common.h:620`; `--help` default), tool/adapter default **8088**; real orphaned unit on this machine omits `--port` |
|
||
|
||
**Root cause chain (one paragraph):** `detect_llama_version` (bin/pos-ai-server:71) always returns "unknown" because llama.cpp's `--version` writes to stderr and the tool discards stderr (`2>/dev/null`). Independently, `validate_default_flags`/`validate_requested_flags` (lines 101, 145) run `printf '%s' "$help_text" | grep -qE ...` under `set -o pipefail`; `grep -q` exits at the first match (flags are at byte offsets 320–39,741 of a 59,000-byte help), the bash-builtin `printf` then hits EPIPE, and pipefail promotes the SIGPIPE (rc=141) — so the `if` is false even though the flag IS in the help. Which flags "fail" each run is a scheduling race (empirically 0–4 flags rejected per run). Separately, `resolve_model` (lines 229-259) rejects the user's directory argument `Qwen-Qwen3-1.7B-GGUF` (HF downloader produces `$HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf`, not a flat file), and the unit write precedes an unguarded `systemctl --user daemon-reload` that fails under SSH with no `XDG_RUNTIME_DIR`, aborting via `set -e` before the linger hint and leaving an orphaned unit whose ExecStart may already have lost `--port` (real example on this machine: `ExecStart=... --host 127.0.0.1` only), guaranteeing a port mismatch against the adapter's 8088 health/API probes.
|
||
|
||
---
|
||
|
||
## Step 1: H1 — Version detection "unknown"
|
||
|
||
### Hypothesis (as briefed)
|
||
`detect_llama_version` (`bin/pos-ai-server:65-74`) greps `[0-9]+\.[0-9]+\.[0-9]+` (semver) from `--version`; llama.cpp uses build numbers `bNNNNN`, never X.Y.Z.
|
||
|
||
### Actual llama.cpp `--version` output (b10822)
|
||
Downloaded and executed the real release binary (`llama-b10822-bin-ubuntu-x64.tar.gz`, `https://github.com/ggml-org/llama.cpp/releases/tag/b10822`):
|
||
|
||
```
|
||
$ ./llama-server --version
|
||
version: 0.4.0-dev (build 10822, commit c457e3bf7)
|
||
built with GNU 11.4.0 for Linux x86_64
|
||
```
|
||
|
||
**The format DOES contain X.Y.Z (`0.4.0`).** The briefed premise is factually wrong for current builds. The regex would match `0.4.0`.
|
||
|
||
### The REAL mechanism — output goes to stderr
|
||
```
|
||
$ ./llama-server --version 1>/dev/null # output STILL appears
|
||
$ ./llama-server --version 2>/dev/null # NOTHING appears
|
||
```
|
||
`od -c` confirms the bytes are written to **stderr**. Source citation: `common/build-info.h:13`
|
||
```cpp
|
||
void llama_print_build_info(const char *, FILE * = stderr);
|
||
```
|
||
called from `common/arg.cpp:1456` (`llama_print_build_info(llama_version()); exit(0);`) — default stream stderr.
|
||
|
||
`detect_llama_version` (`bin/pos-ai-server:71`):
|
||
```bash
|
||
version="$("$bin" --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
|
||
```
|
||
`2>/dev/null` discards the ONLY output → grep gets empty input → `version=` empty → `"unknown"`. Proven against real binary: with `2>/dev/null` captured nothing; with `2>&1` captured `0.4.0`.
|
||
|
||
Note: even with `2>&1`, the regex is brittle against older builds that print only `build 10822` (no semver) — a secondary hazard, not the current blocker.
|
||
|
||
```
|
||
Hypothesis: version always "unknown" because regex can't match
|
||
Why plausible: builds are tag "b10822"
|
||
Evidence supporting: user sees "unknown" in all messages
|
||
Evidence against: actual output has X.Y.Z ("0.4.0"); the regex DOES match
|
||
Test needed: run detect_llama_version verbatim against real binary
|
||
Result: empty capture (stderr discarded) → "unknown"
|
||
Conclusion: CONFIRMED outcome, wrong mechanism — stderr discard, not regex format
|
||
```
|
||
|
||
**[DONE]**
|
||
|
||
## Step 2: H2 — Wrong binary found
|
||
|
||
### (a) What the release archive ships
|
||
`llama-b10822-bin-ubuntu-x64.tar.gz` (downloaded and listed) contains `llama-server` (among ~30 `llama*` tools + `.so` libs). **No bare `server` binary.**
|
||
|
||
### (b) Installer symlink behavior
|
||
`apps/ai/llamacpp.sh:52-55`:
|
||
```bash
|
||
for bin in \$install_dir/llama*; do
|
||
[ -f "\$bin" ] && [ -x "\$bin" ] || continue
|
||
sudo ln -sf "\$bin" /usr/local/bin/\$(basename "\$bin")
|
||
done
|
||
```
|
||
Since the archive ships `llama-server`, the glob `llama*` matches it → `/usr/local/bin/llama-server` IS created. Verified on the target machine (real install): `/usr/local/bin/llama-server -> /usr/local/lib/llama.cpp-b10822/llama-server`, and `llama-server --help` (PATH lookup) returns the full 732-line/59,000-byte help. RUNPATH `$ORIGIN` (`readelf -d`: `Library runpath: [$ORIGIN]`) makes the shared libs resolve through the symlink. **The install is not broken this way — H2 is not the user's cause.**
|
||
|
||
### (c) The bare `server` fallback is a genuine latent defect (fixture)
|
||
`find_llamacpp` (`bin/pos-ai-server:56`) candidates: `llama-server`, `llama.cpp/server`, `server`, `llama-server-cuda`. Fixture: PATH containing ONLY an unrelated `/tmp/h2fixture/bin2/server` (no llama-server):
|
||
|
||
```
|
||
find_llamacpp -> 'server' (command -v -> '/tmp/h2fixture/bin2/server')
|
||
detect_llama_version -> (depends on the unrelated binary)
|
||
validate_default_flags:
|
||
--port: UNSUPPORTED (warn + omit)
|
||
--host: UNSUPPORTED (warn + omit)
|
||
--n-gpu-layers: UNSUPPORTED (warn + omit)
|
||
--ctx-size: UNSUPPORTED (warn + omit)
|
||
--threads: UNSUPPORTED (warn + omit)
|
||
```
|
||
i.e., a wrong-binary scenario produces the same family of user-visible messages — but **not** what this user hit (their install is correct and `llama-server` resolves first because candidate order checks `llama-server` before `server`).
|
||
|
||
```
|
||
Conclusion: REFUTED as user's root cause; PARTIALLY-CONFIRMED as latent defect (bare `server` fallback)
|
||
```
|
||
|
||
**[DONE]**
|
||
|
||
## Step 3: H3 — Help-format regex vs real llama-server --help — **actual root cause of the flag rejections**
|
||
|
||
### The regex matches real help
|
||
Real `llama-server --help` (732 lines) contains (`grep -n` on real output):
|
||
```
|
||
7: -t, --threads N
|
||
25: -c, --ctx-size N
|
||
140: -ngl, --gpu-layers, --n-gpu-layers N
|
||
503: --host HOST
|
||
506: --port PORT
|
||
```
|
||
Word-boundary regex `(^|[[:space:]])${flag}([[:space:]]|=|$)` matches all five when tested WITHOUT pipefail (verified repeatedly). H3 as stated ("format mismatch misses real flags") is **REFUTED**.
|
||
|
||
### The real failure: `printf | grep -q` + `set -o pipefail` → SIGPIPE race
|
||
`bin/pos-ai-server` line 2: `set -euo pipefail`. Lines 101/145:
|
||
```bash
|
||
if printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then
|
||
```
|
||
`grep -q` exits as soon as it finds a match (closing the pipe's read end). The produced help_text is 59,000 bytes; the pipe buffer is 64 KB. Flags appear at byte offsets:
|
||
- `--threads` 320, `--ctx-size` 1892, `--n-gpu-layers` 10992 (early)
|
||
- `--host` 39499, `--port` 39741 (late)
|
||
|
||
For early flags, grep matches and exits after reading ≤ a few KB; the bash-builtin `printf` still has ~57 KB to write → EPIPE → under `pipefail` the pipeline returns **141 (128+13=SIGPIPE)** → the `if` is false even though grep found the flag. For late flags, printf usually completes writing into the 64 KB buffer before grep exits → rc=0. It's a **race**, so outcomes vary run to run.
|
||
|
||
**Isolated proof** (same capture, same regex, pipefail on):
|
||
```
|
||
++ printf '%s' "$help_text" | grep -qE -- '(^|[[:space:]])--ctx-size([[:space:]]|=|$)'
|
||
pipeline rc=141 (grep -q DID match; printf died of SIGPIPE)
|
||
```
|
||
30-run trial: MATCH/NOMATCH alternated ~50/50 for `--ctx-size`.
|
||
|
||
**Live `pos-ai-server` flakiness (10 runs, real binary, identical inputs):**
|
||
```
|
||
run 1: --ctx-size --threads
|
||
run 2: --ctx-size --threads
|
||
run 3: --threads
|
||
run 4: --threads
|
||
run 5: --n-gpu-layers --ctx-size --threads
|
||
run 6: --ctx-size
|
||
run 7: --port --n-gpu-layers --threads
|
||
run 8: --host --n-gpu-layers --ctx-size --threads
|
||
run 9: --n-gpu-layers --ctx-size --threads
|
||
run 10: (none!)
|
||
```
|
||
This explains the user's per-run differences (runs 1-2 vs run 3): **there is no CLI/config/env difference** — `REQUESTED_FLAGS` and `CONFIG_REQUESTED_FLAGS` are empty (no ai.env `LLAMACPP_*` keys, no CLI flags; verified the `requested_from_env_config` calls return early). Flag validation runs identically each time; the outcome is a scheduling race. Both `validate_default_flags` (warn+omit) and `validate_requested_flags` (hard `err`) carry the same bug — a requested `--ctx-size` would randomly hard-fail with "does not expose".
|
||
|
||
Verdict: **H3 REFUTED as stated; SIGPIPE+pipefail race is THE root cause of the flag rejections.**
|
||
|
||
**[DONE]**
|
||
|
||
## Step 4: H4 — Model resolution
|
||
|
||
`resolve_model` (`bin/pos-ai-server:229-259`) accepts: absolute FILE path (`[ -f ]`), `$HF_DOWNLOAD_DIR/<name>` **FILE** (`[ -f "$candidate" ]`), or relative FILE. It never treats a DIRECTORY under `$HF_DOWNLOAD_DIR` as a model.
|
||
|
||
Downloader layout (`bin/pos-ai-hf`): `hf_repo_dir()` (lines 308-311) → `$HF_DOWNLOAD_DIR/${repo_id//\//-}` — e.g. repo `Qwen/Qwen-Qwen3-1.7B-GGUF` → dir `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/` containing `Qwen3-1.7B-Q8_0.gguf` + `.hf-meta`.
|
||
|
||
Reproduction (fixture dir created exactly like the user's; `.gguf` stub inside):
|
||
```
|
||
$ pos-ai-server start Qwen-Qwen3-1.7B-GGUF
|
||
[!] installed llama.cpp unknown does not support default flag --threads — omitting it from the unit
|
||
ERROR: Model not found: Qwen-Qwen3-1.7B-GGUF (also searched /home/unknown/.local/share/linux_post_install/ai/models)
|
||
EXIT: 1
|
||
```
|
||
Exact user message. The user's `mv Qwen3-1.7B-Q8_0.gguf ../` moved the file into `$HF_DOWNLOAD_DIR/`, but the arg `Qwen-Qwen3-1.7B-GGUF` is still a dir name → still fails (run 2 reproduced identically). The absolute file path (run 3) succeeds at model resolution (reproduced).
|
||
|
||
`pick_model` (lines 205-227) uses `find "$HF_DOWNLOAD_DIR" -name '*.gguf' -type f` — RECURSIVE, does NOT miss subdirs (verified). But it is only reached when no explicit arg/config is given; the user passed an explicit arg, so it wasn't involved.
|
||
|
||
**H4 CONFIRMED.**
|
||
|
||
**[DONE]**
|
||
|
||
## Step 5: H5 — dbus/systemctl under SSH
|
||
|
||
Code ordering (`bin/pos-ai-server`):
|
||
- `set -euo pipefail` (line 2)
|
||
- unit write + `chmod 644` (lines 584-602)
|
||
- `systemctl --user daemon-reload` (line 605)
|
||
- `systemctl --user enable --now "$SERVICE"` (line 606)
|
||
- linger hint (lines 610-615)
|
||
|
||
No pre-check of the user bus anywhere in the file. Same pattern in `pos-communication-matrix-listener:341-342` and `pos-network-download:191-192`; NO tool pre-checks the bus (`pos-entertainment-status:55` uses `systemctl --user show-environment` only as a query guard, not an enable guard).
|
||
|
||
Reproduced exactly on target machine (SSH-like shell: `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS` unset):
|
||
```
|
||
$ systemctl --user show-environment
|
||
Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)
|
||
rc=0 (exit status shown as 0 because of head pipe; the systemctl command itself fails)
|
||
```
|
||
With `set -e`, line 605's failure aborts the script; the linger hint (611-615) never runs.
|
||
|
||
Remediation verified: with ONLY `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (dir exists), `systemctl --user show-environment` succeeds. `loginctl enable-linger` is the standard persistence fix.
|
||
|
||
Orphaned-unit side effect confirmed: the unit file remains written even though daemon-reload failed (a real orphaned unit exists at `~/.config/systemd/user/pos-ai-server.service` from the failed attempt on this machine — see Step 6 for its corrupted ExecStart).
|
||
|
||
**H5 CONFIRMED.**
|
||
|
||
**[DONE]**
|
||
|
||
## Step 6: H6 — Port mismatch (downstream impact)
|
||
|
||
llama.cpp default port: **8080**.
|
||
- Binary evidence: real `--help` → `--port PORT port to listen (default: 8080)`
|
||
- Source citation: `common/common.h:620` → `int32_t port = 8080; // server listens on this network port` (also `common/common.h:261` in struct block; `from https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/common.h`)
|
||
|
||
Tool/adapter default: **8088** — `bin/pos-ai-server:329` (`PORT="${LLAMACPP_PORT:-8088}"`), `:281`, `:316`; `lib/ai-providers/llamacpp.sh:15,23,49` (adapter probes `http://$host:$port/v1/models` and `/v1/chat/completions`); health check `bin/pos-ai-server:198` probes `$HOST:$PORT`.
|
||
|
||
Real corrupted unit found on this machine (from the failed start attempt — the SIGPIPE race omitted flags):
|
||
```
|
||
ExecStart="/usr/local/bin/llama-server" -m "…/Qwen3-1.7B-Q8_0.gguf" --host 127.0.0.1
|
||
```
|
||
Only `--host` survived — `--port 8088`, `--n-gpu-layers 0`, `--ctx-size 4096`, `--threads N` were all dropped. If `systemctl` had succeeded, llama-server would bind **8080** while `check_health` and the OpenAI adapter probe **8088** → "not running" / connection refused. Even when the race lets the unit through, H6 guarantees a downstream mismatch whenever `--port` is omitted.
|
||
|
||
**H6 CONFIRMED.**
|
||
|
||
**[DONE]**
|
||
|
||
## Root Cause Statement (final)
|
||
|
||
The user's exact messages trace to four independent defects in `bin/pos-ai-server` (plus one in the adapter default):
|
||
|
||
1. **"installed llama.cpp unknown"** — `detect_llama_version` (`:71`) discards stderr; real llama.cpp b10822 prints `version: 0.4.0-dev (build 10822, …)` to **stderr** (`common/build-info.h:13`), so the capture is always empty → "unknown".
|
||
2. **"does not support default flag …"** — `printf '%s' "$help_text" | grep -qE` under `set -o pipefail` races: grep -q exits at first match, printf gets SIGPIPE, pipefail promotes rc=141, so valid flags are randomly judged "unsupported" and omitted from the unit (or hard-errored when user-requested). This is the reason runs 1-2 and run 3 flagged different sets of flags — pure scheduling, not input differences.
|
||
3. **"Model not found: Qwen-Qwen3-1.7B-GGUF"** — the HF downloader puts weights at `$HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf`, but `resolve_model` accepts only files; a directory argument fails (`:241-248`), even after the user's `mv` (the arg was still a directory name).
|
||
4. **"Failed to connect to user scope bus …"** — the unit write (`:584-602`) precedes an unguarded `systemctl --user daemon-reload` (`:605`); under SSH neither `XDG_RUNTIME_DIR` nor `DBUS_SESSION_BUS_ADDRESS` is set, so systemctl fails, `set -e` aborts, and the linger fix hint (`:611-615`) never shows. The orphaned unit left behind can carry a corrupted ExecStart (see H6), e.g. missing `--port 8088` so the server would bind llama.cpp's default 8080 while the adapter/health check probe 8088.
|
||
|
||
Classification: **FACT** (mechanisms directly reproduced with the real b10822 binary; source citations for stderr stream and port default; unit artifact inspected).
|
||
|
||
## Fix-Point Spec
|
||
|
||
| # | File:line | Defect | Minimal change | Design question for Architect |
|
||
|---|-----------|--------|----------------|-------------------------------|
|
||
| F1 | `bin/pos-ai-server:71` | version always "unknown" (stderr discarded; regex also brittle for pure-build strings) | Capture `2>&1`; broaden regex to also accept `build [0-9]+`/`b[0-9]+`: `version="$("$bin" --version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+' | head -1 …)"` | Canonical display: semver vs build number vs both? |
|
||
| F2 | `bin/pos-ai-server:101,145` | `printf|grep -q` + pipefail SIGPIPE race → random flag rejection | Replace the pipeline: `grep` without `-q` writing to `/dev/null`, e.g. `if grep -E -- "…" <<<"$help_text" >/dev/null; then` (non-q grep consumes all input; herestring avoids the pipe and printf EPIPE), **or** bash regex `[[ "$help_text" =~ (^|[[:space:]])${flag}([[:space:]]|=|$) ]]` | Style preference: grep-herestring vs bash `=~`; whether to harden `validate_requested_flags` identically (yes) |
|
||
| F3 | `bin/pos-ai-server:229-259` (`resolve_model`) | directory under `HF_DOWNLOAD_DIR` (one `.gguf`) rejected | When `$candidate` is a directory: if exactly one `*.gguf` inside → use it; if multiple → list and err/ask | Should auto-expand single-gguf dirs, or require explicit file path? (Recommended: auto-expand, since downloader always produces `<repo>/<file>` layout) |
|
||
| F4 | `bin/pos-ai-server:605-606` (+ same in matrix-listener:341, network-download:191) | no user-bus pre-check; SSH w/o XDG_RUNTIME_DIR → set -e abort; orphan unit; linger hint skipped | Pre-flight before daemon-reload: `systemctl --user show-environment` (or `printenv XDG_RUNTIME_DIR DBUS_SESSION_BUS_ADDRESS`); on failure `err` with remediation: `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if dir exists) and `sudo loginctl enable-linger $(id -un)`; optionally run the server directly (no unit) as fallback | Fallback strategy: error+hint only, or run-direct fallback? Also: consider removing the orphaned unit on failure (Builder decision) |
|
||
| F5 | `bin/pos-ai-server:56` (`find_llamacpp`) | bare `server` / `llama.cpp/server` fallbacks can pick unrelated binaries | Drop `"server"` (keep `llama-server`, `llama-server-cuda`; keep or drop `llama.cpp/server`) | Should the fallback list be `llama-server`/`llama-server-cuda` only? |
|
||
| F6 | `bin/pos-ai-server:528-532` + `lib/ai-providers/llamacpp.sh:15,23,49` | `--port` omission → server on 8080 vs adapter/health on 8088 | Guarantee `--port $PORT` always emitted (F2 fixes the omission); as defense-in-depth, adapter health fallback probe both 8088 and 8080, or derive from the unit | Should the tool ever allow running on llama.cpp's default 8080, or always pin 8088? |
|
||
| F7 | `apps/ai/llamacpp.sh:46-57` | installer never verifies the installed binary runs (version/help readable) | Post-install sanity: `llama-server --version >/dev/null 2>&1 && llama-server --help >/dev/null`; warn on failure | None (simple hardening) |
|
||
|
||
## Design Questions (explicit)
|
||
|
||
1. **F2/F3 boundary:** Should unknown-version disable flag *validation* entirely (trust defaults), or retain help-based validation but fix the SIGPIPE race? (Recommended: keep validation, fix race — help-based validation is the correct design once deterministic.)
|
||
2. **F3:** Should `resolve_model` auto-expand a single-`.gguf` directory; with multiple gguFs, err with the list?
|
||
3. **F4:** On unresolvable user bus: error+hint only, or a run-direct fallback (no systemd unit) for headless/SSH use?
|
||
4. **F5:** Drop the bare `server` (and possibly `llama.cpp/server`) fallback candidates?
|
||
5. **F6:** Pin the port in the unit always (recommended) vs teach the adapter to probe the llama.cpp default 8080 as a fallback?
|
||
|
||
## Test-Fixture Spec (for Tester)
|
||
|
||
Each fix gets a stub/PATH/case that reproduces it deterministically:
|
||
|
||
- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit c457e3bf7)` **to stderr**, help to stdout. Assert `detect_llama_version` returns `0.4.0` (not "unknown") with the fixed `2>&1`; also fixture printing only `version: b10822`/`build 10822` to stderr to assert the broadened regex.
|
||
- **F2 (race):** fake `llama-server` whose `--help` emits a 59 KB body with tokens `--threads` at line 7, `--ctx-size` at line 25, `--n-gpu-layers` at line 140, `--host`/`--port` near the end (exact real llama.cpp layout). Run `validate_default_flags` 20× under `set -euo pipefail`; fixed code must report all 5 supported on EVERY run (deterministic). Regression tail: pre-fix, the run must fail at least once (demonstrates the race existed).
|
||
- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `pos ai server start Qwen-Qwen3-1.7B-GGUF` (DRY_RUN=1) resolves to the file; multi-gguf dir case errors with the file list.
|
||
- **F4 (bus):** environment with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or a stub `systemctl` that fails with the dbus message); assert the pre-flight fails with the remediation hint AND the linger hint text when applicable; assert no orphaned unit is left (or is removed on failure).
|
||
- **F5 (server fallback):** PATH containing ONLY a fake unrelated `server` (prints its own --help/--version) and no llama-server; assert `find_llamacpp` does NOT return `server`.
|
||
- **F6 (port):** with F2 fixed, generate a unit with `DRY_RUN=1` and assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`).
|
||
- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (e.g., missing shared lib) → assert post-install sanity warns.
|
||
|
||
## Evidence Index (concrete artifacts)
|
||
|
||
- Real binary downloaded/run: `llama-b10822-bin-ubuntu-x64.tar.gz` from `https://github.com/ggml-org/llama.cpp/releases/tag/b10822` (asset `llama-b10822-bin-ubuntu-x64.tar.gz`).
|
||
- `llama-server --version` → stderr: `common/build-info.h:13` (`FILE * = stderr`), `common/arg.cpp:1454-1458`, plus byte-level `od -c`/redirect proof.
|
||
- `--help` flag lines: real output lines 7/25/140/503/506; byte offsets 320/1892/10992/39499/39741; total 59001 bytes (< 64 KB pipe buffer).
|
||
- Pipeline rc=141 proof: `/tmp/test_sigpipe3.sh` output (rc 0/141 alternating).
|
||
- 10-run `pos-ai-server` flakiness table (see Step 3).
|
||
- Real corrupted orphaned unit: `~/.config/systemd/user/pos-ai-server.service` — `ExecStart=… --host 127.0.0.1` (port/ctx/threads/gpu-layers omitted).
|
||
- Model-not-found exact reproduction and post-`mv` reproduction.
|
||
- User-bus error reproduced verbatim on target machine; `export XDG_RUNTIME_DIR=/run/user/$(id -u)` remediation verified working.
|
||
- Default port: `common/common.h:620` (`int32_t port = 8080`); real `--help` `(default: 8080)`.
|
||
- H2 fixture: fake unrelated `server` → `find_llamacpp` returns `server`, all defaults "unsupported".
|
||
|
||
## Next Agent
|
||
|
||
**Builder** — five (F1-F5) of the six fix-points are code changes in `bin/pos-ai-server` (plus `apps/ai/llamacpp.sh` for F7); Architect input needed on the design questions before/while implementing F2/F3/F4/F6.
|
||
|
||
**Changes made by Detective:** none (read-only; only downloaded/extracted to /tmp and created user-level fixtures under `/home/unknown/.local/share/linux_post_install/ai/models` matching the user's layout for reproduction). |