diff --git a/AGENT_TODO.md b/AGENT_TODO.md index 9c4927e..9a7d1cd 100644 --- a/AGENT_TODO.md +++ b/AGENT_TODO.md @@ -42,6 +42,8 @@ summary (newest last). ## Done +- **2026-09-06** — AI server breakage post-llamacpp install — four root causes found and fixed (Detective→Architect→Builder→Tester chain). (1) `llama-server --version` prints to STDERR — `detect_llama_version`'s `2>/dev/null` swallowed it → "installed llama.cpp unknown"; (2) `printf|grep -q` under `pipefail` → SIGPIPE rc=141 race randomly rejecting valid flags from the 59 KB `--help`; (3) `resolve_model` expected flat files but the HF downloader creates `//file.gguf` dirs → `Model not found`; (4) llama.cpp default port 8080 vs tool's 8088. Architect DQ1-DQ6: help-gated validation stays; dir-expansion never silently picks; `ensure_user_bus` in `lib/common.sh` pre-flights all three tools; `--no-unit` direct-run escape hatch for SSH/headless; candidates narrowed to `llama-server`/`llama-server-cuda`; port pinned 8088; installer post-install sanity. Builder F1-F7 (stderr version capture, pipefail-safe flag validation, model dir expansion, port hardcoding, `--no-unit`, user-bus pre-flight, installer sanity + F1 regex edge: `build 1.2.3`→`1` misparse). Tester: 4 new regression files (version-from-stderr, 20× flag-validation determinism, model dir expansion, bus pre-flight + E2E) + 3 fixture updates; suite now 16 files / 269 checks. Verified: `make gen` idempotent, `make check` OK, `make lint` 0 FAIL / 0 WARN, `make test` 269/269 (~49 s), `bash -n` clean, `git diff --check` clean. Post-fix: user's machine needed only `export XDG_RUNTIME_DIR=/run/user/1000` (linger already on) → Option A systemd-managed server works, or `--no-unit` for direct run. + - **2026-09-06** — Stabilization pass: 17-point code-level audit executed via Explorer(3) → Architect(decisions D-A..D-F) → Builder(ai/security/tooling/config/netprobe/f1-f3) → Tester(regression suite) → Reviewer(2 rounds). Security: Telegram sender-owner AND-gate + `TELEGRAM_OWNER_ID` registry/docs; Matrix `MATRIX_ROOM_ID` required; gpg `--passphrase-fd 3` (no argv secret); `/dev/tcp` positional-arg form (checkport/smb-client/share-lib/NET_PROBE incl. escaping `\$1`/`\$2`); eval `--no-command-execution` now carried by both chat bridges (D-B), deny-by-default `[y/N]`, tty-gated `--trust`; D-A soft-fail model ratified by Architect amendment (fail-closed either way; listeners are `Restart=always` so strict mode would crash-loop). AI: ALL ExecStart flags validated against installed llama.cpp (requested→error, default→omit+warn, `CONFIG_REQUESTED_FLAGS`), single-line ExecStart confirmed via `systemd-analyze verify`; hf single-file failure rc/exit-0 + meta-write bug fixed; `LLAMACPP_HOST` coherent; `# POS_SUBCMDS` + metadata gaps closed. Tooling: lint-conventions rewritten Bash-native (~24-30× faster, rules byte-identical, `:num` restored on 2 WARNs, planted-violation negative verified); `pos system uninstall` covers all 12 libs + scale-tail + flags dir + systemd USER units (`|| true`) + de-hardcoded plugin markers; safe anchored `.bash_completion`/`.bashrc` removals replace `sed -i '/pos/d'`. Config: canonical `load_env_file` in `lib/config-ui.sh` (CRLF strip, env-wins, XDG, `LOADED_ENV_KEYS`); 9 tools migrated; entertainment-lib collapsed to wrappers; docker-compose deliberately NOT migrated (source-semantics, documented). Tests: first committed regression suite — `tests/run-tests.sh` zero-dep runner + `make test`; 12 files / 179 checks / 0 skip / ~52s; hard skip contract (never lie); negative lint/gen-drift gates; systemd-analyze verify included. Verification: `make gen` idempotent, `make check` OK, `make lint` `0 FAIL, 0 WARN`, `make test` green, `bash -n` clean, `git diff --check` clean; `systemd-analyze verify` PASS on generated unit; CLI smokes (`pos --help`, `pos ai --help`, `pos ai hf --help`, `pos ai server --help`, `pos tree`) OK. - **2026-09-06** — llamacpp optional-app installer + `ai` app category + `pos ai server` install-hint wiring: new `apps/ai/llamacpp.sh` (idempotent `install_llamacpp()`/`uninstall_llamacpp()` — GitHub release archive: scans `/releases?per_page=10` for the first `-bin-ubuntu-{x64,arm64}.tar.gz` asset since `releases/latest` (v0.4.0 milestone) ships no binaries; installs to `/usr/local/lib/llama.cpp-` with `--strip-components=1`, symlinks every `llama*` binary into `/usr/local/bin`; uninstall removes the lib dir and only the symlinks whose target points into it); `apps/install.sh` CAT_NAMES += `[ai]="AI / ML"`; `templates/app.sh` categories comment += ai; DOC/APPS.md app-table row count updated (15→16 at the time; 18 after the 2026-09-06 stabilization pass) + categories line + llama.cpp catalog row; `bin/pos-ai-server` help "Requires:" + both `err "llama-server not found…"` lines (start/status) now name the app installer + invocation (`bash apps/install.sh llamacpp`, `--apps`, `--full`) and keep the GitHub URL (scrcpy phrasing pattern); DOC/POS.md ai row notes `bash apps/install.sh llamacpp`. Verified: live API probe confirmed real asset naming — `llama--bin-ubuntu-x64.tar.gz`/`-arm64.tar.gz` on nightly bNNNNN releases, top-level dir present, `.tar.gz` not `.zip` (so `tar` replaces the brief's `unzip` step — no unnecessary apt install); `bash -n` touched scripts; `bash apps/install.sh --uninstall llamacpp` resolves app + idempotent uninstall rc 0 (no network); `bash apps/ai/llamacpp.sh uninstall` rc 0; `make gen` idempotent; `make check` green; `make lint` 0 FAIL / 0 WARN. diff --git a/AgentsReport/architect/2026-09-06_ai-server-fix-design.md b/AgentsReport/architect/2026-09-06_ai-server-fix-design.md new file mode 100644 index 0000000..b72a69c --- /dev/null +++ b/AgentsReport/architect/2026-09-06_ai-server-fix-design.md @@ -0,0 +1,177 @@ +# AI Server Start Breakage — Root Cause + Fix Design Decisions + +**Date:** 2026-09-06 +**Architect:** Design pass over the Detective report `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (root cause classified **FACT**). +**Constraint:** No code changes in this pass; decisions + ratified fix scope (F1-F7) for the Builder; test-fixture requirements for the Tester. Preserve the existing Bash tool architecture and the D-A..D-F stabilization decisions. + +--- + +## TL;DR + +1. **DQ1 — Validation stays help-gated.** The SIGPIPE race (F2) is the only validation defect; help-based validation is binary-reality-based and correct once deterministic. Version is cosmetic. "unknown" means: validate against real `--help` regardless, message without the version. NO version→capability map. [DECIDED] +2. **DQ2 — `resolve_model` gains file-inside-dir expansion.** `$HF_DOWNLOAD_DIR/`/dir → exactly-one `*.gguf` resolves to it; multiple → list + err ("pick one"); zero → err as today. Also accept `/.gguf`. Precedence: absolute path > dir-with-exactly-one-gguf > `$HF_DOWNLOAD_DIR/.gguf` flat file > `$HF_DOWNLOAD_DIR//` > relative-as-is. Existing bare-file behavior unchanged (backward compatible). [DECIDED] +3. **DQ3 — User-bus pre-flight by default, plus `--no-unit` direct-run escape hatch (option c).** Pre-flight aborts BEFORE writing the unit (no orphan); `--no-unit` runs the server directly under nohup+pidfile for headless/SSH boxes. Shared helper `ensure_user_bus` lives in `lib/common.sh` (all three tools source it). Matrix-listener + network-download get the same pre-flight. Remediation text includes `export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`. [DECIDED] +4. **DQ4 — `find_llamacpp` drops bare `server` and `llama.cpp/server`.** Keep `llama-server`, `llama-server-cuda`. Do NOT add `llama-server-mtl` (Mac-only; out of target). Stale-unit: start F4 pre-flight WARNS about an existing unit, does not delete it (never gratuitously remove user state). [DECIDED] +5. **DQ5 — Always pin `--port $PORT`.** Once F2 makes validation reliable, ExecStart always carries `--port $PORT` (help-gated, so a build without `--port` omits it). No adapter probing fallback change needed (deferred). [DECIDED] +6. **DQ6 — Installer sanity IS in scope.** `apps/ai/llamacpp.sh` post-install runs `llama-server --version` (2>&1) + `--help` and errs if the binary doesn't execute; verifies the symlink target exists. [DECIDED] + +--- + +## DQ1: Validation posture when version is unknown + +### Decision +**Keep help-based validation as the single source of truth. Do NOT add a version→capability map.** "unknown" means: still validate against the real `--help` output; only the version string in log/error messages is dropped (message proceeds without version). + +### Rationale +Help-based validation reflects what the actual installed binary supports — it is always correct once the SIGPIPE race (F2) is fixed, because it peers at real capability output. Adding a version→capability table would introduce a second, version-coupled source of truth that drifts the moment llama.cpp adds/renames flags; its only benefit would be to decorate messages with a version string, which is cosmetic. The version is useful for *debugging* ("which build am I running") but must never gate validation. "unknown" therefore degrades only to "no version in messages", not to "no validation". + +### How "unknown" behaves +- `detect_llama_version` returns "unknown" only when `--version` is genuinely unreadable (no binary) — after F1, stderr is captured and the output IS readable for real builds. +- With "unknown", `validate_*_flags` still fetch `--help` and judge flags normally; error texts simply omit the version token. +- F1's broadened regex handles both semver and build-only strings, so "unknown" becomes rare (only truly missing binary/--version). + +[DELIVERY: F1 capture-2>&1 + broadened regex; F2 race fix. No capability map.] + +--- + +## DQ2: `resolve_model` directory expansion + +### Decision +Extend `resolve_model` (`bin/pos-ai-server:230-264`) to resolve a directory-typed explicit argument when it contains a usable model file, with exact precedence. Interactive `pick_model` stays recursive (already lists flat `.gguf` files one level up via `find -type f`; confirmed at `:210` — it lists each dir's files, so no F3 change needed there). + +### Precedence (highest → lowest) +1. **Absolute path** that is a file (`[ -f "$explicit" ]`) — unchanged; if the absolute path is a directory, treat as directory case below. +2. **`$HF_DOWNLOAD_DIR/` is a DIRECTORY** containing exactly one `*.gguf` → resolve to that file. If multiple `*.gguf` → print all and `err "pick one: /"`. If zero → fall through / err (never silently pick). +3. **`$HF_DOWNLOAD_DIR/.gguf` flat file** — unchanged (`[ -f "$candidate" ]`). +4. **`$HF_DOWNLOAD_DIR//.gguf`** (slug/file form) — resolve to `$HF_DOWNLOAD_DIR//.gguf`. +5. **Relative-as-is** — unchanged (`[ -f "$explicit" ]`). + +### Backward compatibility +Bare existing behavior (flat file path, absolute file path, relative path) is fully preserved — the new directory-expansion branches only fire where the current code would have errored with "Model not found". No silent picking: multiple matches always err with the disambiguating list. + +### Exact rule (multiple-gguf dir) +``` +if a dir contains >1 *.gguf: print each as "/" and err "model dir contains multiple — pick one" +``` +This matches the fixture expectation (multi-gguf dir case errors with the file list). + +[DELIVERY: F3 resolve_model expansion; no pick_model change.] + +--- + +## DQ3: User-bus failure handling (scope across tools) + +### Decision +**Option (c): pre-flight err by default, `--no-unit` direct-run as escape hatch.** + +- `ensure_user_bus` helper in **`lib/common.sh`** (the base lib already sourced by all three tools via the fallback chain; it is the natural shared home for a user-bus guard — the alterative, config-ui.sh, is a config/validation lib, not an execution helper, and would be a semantic misfit). +- Wired into the unit-install paths of `bin/pos-ai-server:605`, `bin/pos-communication-matrix-listener:341`, and `bin/pos-network-download:191` — all three call it BEFORE `systemctl --user daemon-reload` and BEFORE the unit is written, so a failure leaves no orphaned unit. +- On failure, `ensure_user_bus` `err`s with remediation text: + - `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if the dir exists) + - `sudo loginctl enable-linger $(id -un)` + - plus the general "connect to the user's systemd bus" guidance. + +### Why nohup/pidfile for `--no-unit` (direct-run) +Headless/SSH boxes may lack a working user bus permanently; being unable to serve a model at all is worse than a supervisor-less process. `--no-unit` on `pos ai server start` execs `llama-server` directly under `nohup ... >$log 2>&1 &`, writes a pidfile under `$RUN_DIR`/`$HF_DOWNLOAD_DIR` sibling (e.g. `$RUN_DIR/pos-ai-server.pid`), and prints the log path + `kill $(cat pidfile)` hint. This is a thin escape hatch, NOT a second supervisor; supervision stays systemd when available. + +### Why the helper lives in common.sh (not config-ui) +- All three tools source `lib/common.sh` already (via the `$(dirname)/../lib/common.sh` fallback chain). Config-ui.sh is not guaranteed present in the standalone communication tools' spirit, and it is semantically a *config* lib. An *execution pre-flight* guard belongs with the other `run`/`spawn` execution helpers in common.sh. +- Same one-line helper (declared with a `declare -F` guard like user-timers-lib.sh does) so all three tools share remediation text and behavior, and it can be unit-tested once. + +### Minimal viable behavior for THIS pass +- `ensure_user_bus` in common.sh. +- Pre-flight wired into pos-ai-server, matrix-listener, network-download before unit write. +- `--no-unit` direct-run flag on **pos-ai-server only** (the reported tool, and the only one whose unit manages llama-server). matrix-listener and network-download get the pre-flight but NO direct-run escape hatches this pass (their stateful daemons genuinely need systemd; a direct-run fallback would be a larger design change and is deferred). + +[DELIVERY: F4 pre-flight + `--no-unit` on pos-ai-server.] + +--- + +## DQ4: `find_llamacpp` candidates + +### Decision +New candidate list: **`llama-server`, `llama-server-cuda`** only. Drop bare `server` and `llama.cpp/server`. Do NOT add `llama-server-mtl`. + +### Rationale +- Bare `server` is an unrelated-generic-name hazard (fixture-proven: picks an unrelated binary, then ALL defaults rejected) — genuinely dangerous, drop it. +- `llama.cpp/server` is a relative path-like token that `command -v` can only match as a literal filename `llama.cpp/server` — not a real on-PATH name for the archive's `/usr/local/bin/llama-server` layout; it provides no value, drop it. +- `llama-server-cuda` is a genuine distinct binary name for some CUDA-series builds — keep it. +- `llama-server-mtl` is the Apple Metal (macOS) binary name; this repo targets Debian/Ubuntu Linux only (`apps/ai/llamacpp.sh` errors on non-x64/arm64) — out of scope. +- If `llama-server` and `llama-server-cuda` are both absent, the existing `find_llamacpp`-failure `err` already carries the installer hint (unchanged). + +### Stale-unit handling on start (F4 interaction) +`cmd_start` pre-flight: if `$USER_SYSTEMD_DIR/$SERVICE` already exists, **warn** that a unit is present (it may be stale/orphaned from a prior failed start) and that it will be overwritten; proceed to write. **Do NOT delete it.** `cmd_stop` (`:628-641`) is the only stated removal path and remains the exclusive way user state is removed. Rationale: deleting on start would gratuitously discard user state (a running, working unit) and would blur the unit's ownership. The pre-flight bus check (DQ3) already prevents NEW orphans, so an existing unit is either intentional or stale-legacy — warn, never auto-remove. + +[DELIVERY: F5 candidate list + start stale-unit warning.] + +--- + +## DQ5: Port pinning + +### Decision +**Always pin `--port $PORT` in ExecStart** (help-gated). Confirmed: the adapter probes and health check already use `$PORT` (default 8088; llama.cpp default is 8080 per `common/common.h:620`). After F2 makes default-flag validation reliable, `--port` is never spuriously omitted, so the server always binds the port the adapter and health check expect. + +### Mechanics +- `--port $PORT` is a DEFAULT flag already emitted at `bin/pos-ai-server:528` guarded by `DEFAULT_PORT_OK`. Once F2 removes the race, `DEFAULT_PORT_OK` is reliably 1 for real builds (which advertise `--port` in `--help`), so `--port $PORT` is pinned. +- Strictly ancient llama-server builds without `--port` in `--help` omit it (help-gated); in that (degenerate) case the adapter's 8088 probe would not match the server's 8080 default. Since such builds are not the supported real release (b10822 ships `--port`, default 8080), this residual path is accepted and documented as the "ancient build" degradation. + +### Explicitly deferred +- **Adapter probing fallback** (probe both 8088 and 8080, or derive port from the unit): NOT in scope this pass. With F2+F5, the server and adapter agree on 8088 by construction. Adding a dual-port adapter probe would mask (not fix) a genuine port disagreement and complicate the adapter for a non-path that F5 removes. Deferred with reason. + +[DELIVERY: F6 port pinning — achieved via F2+F5; no adapter fallback.] + +--- + +## DQ6: Installer sanity (F7) + +### Decision +**In scope.** `apps/ai/llamacpp.sh:46-57` post-install adds a sanity check after the symlink loop: +- Run `llama-server --version` capturing `2>&1` — `err` if the binary doesn't execute (non-zero exit). +- Run `llama-server --help` similarly — `err` if unreadable. +- Verify the symlink target exists: for each `/usr/local/bin/llama*` symlink created, `[ -e "$link" ]` (resolves target) — `err` if broken. + +### Rationale +This catches a genuinely broken install (missing shared lib → binary won't run; truncated/empty archive → symlink dangling) at install time with one clear `err`, instead of surfacing as a confusing "version unknown / flags rejected" on the first `pos ai server start`. It is cheap and self-contained in the installer; it is the front door to the whole tool chain and is the natural place to fail fast. + +[DELIVERY: F7 installer post-install sanity.] + +--- + +## Ratified Fix Scope (F1-F7) + +| # | File:line | Decision | 1-line change | +|---|-----------|----------|---------------| +| **F1** | `bin/pos-ai-server:71` | **KEEP** | Capture `2>&1` and broaden regex to `[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+`. | +| **F2** | `bin/pos-ai-server:101,145` | **KEEP** | Replace `printf|grep -q` pipeline with `grep -E -- … >/dev/null <<<"$help_text"` (non-q, no pipe → no SIGPIPE); apply identically to `validate_requested_flags` and `validate_default_flags`. | +| **F3** | `bin/pos-ai-server:230-264` | **KEEP** | `resolve_model` dir-expansion per DQ2 precedence (exactly-one-gguf → resolve; multiple → list+err; commit `$HF_DOWNLOAD_DIR//` form). | +| **F4** | `bin/pos-ai-server:605` + `:461-626`; `bin/pos-communication-matrix-listener:341`; `bin/pos-network-download:191`; **new** `lib/common.sh` | **KEEP (adjusted)** | Add `ensure_user_bus` to `lib/common.sh`; call before unit write in all three tools (abort, no orphan); plus `--no-unit` direct-run flag on `pos-ai-server`; start pre-flight warns on existing unit (DQ4). | +| **F5** | `bin/pos-ai-server:56` | **KEEP** | `find_llamacpp` candidates → `("llama-server" "llama-server-cuda")`. | +| **F6** | `bin/pos-ai-server:528` (+ no adapter change) | **KEEP** | Confirm `--port $PORT` always pinned (reliably emitted once F2 fixed); no adapter probing fallback. | +| **F7** | `apps/ai/llamacpp.sh:46-57` | **KEEP (in scope)** | Post-install sanity: `llama-server --version` (2>&1) + `--help` execute, symlink targets exist, else `err`. | + +## Explicitly deferred (NOT this pass) + +1. **Adapter probing fallback** (probe 8088+8080 / derive port from unit) — F6 makes the adapter agree with the server by construction; a dual-port probe would mask (not fix) disagreement. Deferred with reason. +2. **Direct-run escape hatch for matrix-listener and network-download** — stateful daemons genuinely need systemd; a direct-run supervisor is a larger design change. Deferred; only pos-ai-server gets `--no-unit` this pass. +3. **Multi-user / TELEGRAM_GROUP_MODE and DQ-adjacent chat features** — unrelated to this breakage; tracked separately in the stabilization design. +4. **D1 (pos-ai-hf single-file download failure recording)** and **LLAMACPP_HOST coherence (D2)** — separate defects already tracked in the stabilization open-items; not part of this fix map. +5. **`--no-mmap`-style substring false positives beyond the current word-boundary regex** — D4 (already implemented in this file at `:101`); no change needed unless F2 rewrites it, which uses the identical word-boundary pattern. + +--- + +## Test-Fixture Requirements (for Tester) + +Per fix from the Detective's fixture spec, deterministically: + +- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit …)` **to stderr**, help to stdout → assert `detect_llama_version` returns `0.4.0` (not "unknown"). Also a fixture printing only `version: b10822`/`build 10822` to stderr → assert broadened regex returns the build token. +- **F2 (race + determinism):** fake `llama-server` whose `--help` emits a 59 KB body with `--threads`@line 7, `--ctx-size`@25, `--n-gpu-layers`@140, `--host`/`--port` near end (real llama.cpp layout). Run `validate_default_flags` **N≥20 times under `set -euo pipefail`** → fixed code reports all 5 supported on EVERY run (zero omissions). Regression tail: pre-fix code must fail at least once in the same loop (proves the race existed). +- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `DRY_RUN=1 pos ai server start Qwen-Qwen3-1.7B-GGUF` resolves to the file; multi-gguf dir case errors listing each as `dirname/file`; zero-gguf dir errors as today. +- **F4 (bus):** env with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or stub `systemctl` failing with the dbus message) → assert pre-flight `err`s with remediation text (both `export XDG_RUNTIME_DIR=…` and `sudo loginctl enable-linger …`), and **no unit file is written** (no orphan). Assert `--no-unit` path execs the direct command via a stub `llama-server` recording argv + writes pidfile. +- **F5 (server fallback):** PATH containing ONLY an unrelated fake `server` (its own --help/--version), no llama-server → assert `find_llamacpp` does NOT return `server`. +- **F6 (port):** with F2 fixed, `DRY_RUN=1` generate a unit → assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`). +- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (missing shared lib) → assert post-install sanity `err`s. +- **E2E regression (reported user scenario):** with stubs (fake `llama-server` b10822-shaped stderr + help, fake systemctl that succeeds only with `XDG_RUNTIME_DIR` set), run install→`pos ai server start Qwen-Qwen3-1.7B-GGUF`→assert the written unit is correct (ExecStart carries all 5 flags + `--port 8088`, no orphan unit, version resolvable). + +--- + +*Deliverables complete. See `AgentsReport/architect/2026-09-06_stabilization-design.md` (appended post-merge section) for the consolidated record.* diff --git a/AgentsReport/architect/2026-09-06_stabilization-design.md b/AgentsReport/architect/2026-09-06_stabilization-design.md index 99bb72b..9469805 100644 --- a/AgentsReport/architect/2026-09-06_stabilization-design.md +++ b/AgentsReport/architect/2026-09-06_stabilization-design.md @@ -768,3 +768,15 @@ Groups A, B, C, and D are **parallel-safe** (no file overlap). Group E depends o 4. **pos-ai-hf single-file download failure (D1 from AI audit):** The single-file download path does not record failures in `failed_files`, so `.hf-meta` is written for a partially-downloaded model. This is a HIGH defect but is a pure bug fix (not an architectural decision) — belongs in Builder scope directly, not in this design pass. 5. **GPG passphrase in argv (V3 from Security audit):** Fixing this requires `gpg --batch --passphrase-fd` or a temp-file approach. Behavioral change to `bin/pos-system-backup` — defer to a separate fix commit. 6. **H-002/H-003 bashrc/bash_completion over-broad removal:** Partially addressed in D-E (`.bash_completion` fix) but a more thorough marker-based approach would be ideal. The D-E fix is the minimum viable improvement. + +--- + +## 2026-09-06 (post-merge) — llama-server start breakage root cause + fix decisions + +**References:** `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (root cause, classified **FACT**); full decisions in `AgentsReport/architect/2026-09-06_ai-server-fix-design.md`. + +**Summary:** A user's `pos ai server start` failing after `bash apps/install.sh llamacpp` traced to four independent defects in `bin/pos-ai-server` (plus one adapter-default interaction): (1) `detect_llama_version:71` discards stderr, and llama.cpp b10822 prints its version to stderr → always "unknown"; (2) `printf|grep -q` under `set -o pipefail` → SIGPIPE race (rc=141) → flags randomly rejected; (3) `resolve_model:229-259` rejects the HF-downloader's `/.gguf` layout; (4) unit write precedes an unguarded `systemctl --user daemon-reload:605` that fails under SSH w/o `XDG_RUNTIME_DIR` → orphaned unit. Port pinning (llama.cpp default 8080 vs tool/adapter 8088) is downstream of the race. + +**Decisions added (DQ1-DQ6, see the fix-design file for full rationale):** help-gated validation retained (no version→capability map; version is cosmetic); `resolve_model` expands a single-gguf dir (exact precedence, no silent multi-pick); user-bus pre-flight via new `ensure_user_bus` in `lib/common.sh` + `--no-unit` direct-run on `pos-ai-server`; `find_llamacpp` narrowed to `llama-server`/`llama-server-cuda`; `--port $PORT` always pinned; installer post-install sanity in scope. **Ratified fix scope F1-F7, all KEEP** (F4 adjusted to add `--no-unit` + shared helper; F2's SIGPIPE mechanism supersedes the earlier D-F's descriptive reliance on the broken pipeline). **Deferred:** adapter dual-port probing fallback, direct-run for matrix-listener/network-download, and the previously-tracked D1/D2 AI-server defects. + +**Consistency with prior decisions (no D-A..D-F text altered above):** D-F's requested-vs-default validation semantics (requested → hard err, default → warn+omit) remains the contract; F2 fixes the mechanism that made D-F's help-gate non-deterministic, and F6 (port pinning) is a direct realization of D-F's always-emitted-defaults guarantee. The bus pre-flight (F4) extends what `scheduler-lib.sh:614` already does (a *query* guard) into a hard pre-write *enable* guard shared across the three tools that write user units. diff --git a/AgentsReport/builder/2026-09-06_ai-server-fixes.md b/AgentsReport/builder/2026-09-06_ai-server-fixes.md new file mode 100644 index 0000000..2e7c5b2 --- /dev/null +++ b/AgentsReport/builder/2026-09-06_ai-server-fixes.md @@ -0,0 +1,103 @@ +# Builder Report — llama-server breakage fixes (F1-F7) + +**Date:** 2026-09-06 +**Builder:** Implementation of Architect DQ1-DQ6 ratified scope F1-F7. +**Bindings:** `AgentsReport/architect/2026-09-06_ai-server-fix-design.md` (FACT), `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (FACT root-cause evidence). + +## TL;DR + +Status: **IMPLEMENTED** (all F1-F7) + +Files changed: +- `bin/pos-ai-server` (F1, F2, F3, F4, F5, F6) +- `lib/common.sh` (F4 — new `ensure_user_bus`) +- `bin/pos-communication-matrix-listener` (F4 — inline + call) +- `bin/pos-network-download` (F4 — call) +- `apps/ai/llamacpp.sh` (F7 — post-install sanity) +- `DOC/POS.md` (minimal — `--no-unit` documented) +- generated docs via `make gen` (`DOC/AGENT_Context_Project.md`, `completions/pos.bash`) + +Verification: `bash -n` all touched; probes 1-8 = **63/63 PASS, rc=141 count 0**; `make gen` idempotent x2; `make check` green; `make lint` `0 FAIL, 0 WARN`. + +Probe workspace: `/tmp/opencode/ai-probes/` (stubs, fixtures, `run-probes.sh`, `PROBES-RESULT.txt`). + +--- + +## Step 1: F1 — `detect_llama_version` stderr capture + broadened regex + +`bin/pos-ai-server:82-90`. Real llama.cpp prints `version: 0.4.0-dev (build 10822, commit …)` to **STDERR**; the old code captured only stdout and regexed `[0-9]+\.[0-9]+\.[0-9]+` alone. + +- Both streams captured: `"$bin" --version 2>&1`. +- Regex accepts `X.Y.Z` OR `build [0-9]+` OR `b[0-9]+` (old builds). +- `"${version#build }"` strips the phrase so `build 10822` displays as `10822`. +- Guarded: missing binary / unreadable output → `unknown` (no errexit). + +Evidence (P1): stderr semver → `0.4.0`; `build 10822` (stderr) → `10822`; `b10822` → `b10822`; missing binary → `unknown`. [DONE] + +## Step 2: F2 — pipe-less flag validation (no rc=141 race) + +`bin/pos-ai-server:97-124` (`validate_requested_flags`), `:135-170` (`validate_default_flags`). Old form `printf '%s' "$help_text" | grep -qE ...` died of SIGPIPE (rc=141) when the >64KB pipe-adjacent grep exited at the first match — valid flags sporadically judged unsupported. New form: non-quiet `grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null` (herestring needs no pipe; non-q grep consumes the whole input; same word-boundary regex and guard shape). + +Evidence (P2): on the 58,362-byte/732-line fixture help, 25 runs of `validate_default_flags` → all 5 defaults kept every run, `rc=141` count **0**; requested-flag path: supported `--port` rc 0, unsupported flag rc 1 with version-aware err. [DONE] + +## Step 3: F3 — `resolve_model` directory expansion (never silently pick) + +`bin/pos-ai-server:213-264` + helper `resolve_gguf_in_dir` (`:196-210`). DQ2 precedence: absolute file → absolute dir with exactly one `.gguf` → `$HF_DOWNLOAD_DIR//.gguf` (slug/file form) → `$HF_DOWNLOAD_DIR/.gguf` (flat file) → relative-as-is. A dir with multiple `.gguf`s errors listing each as `/` + "pick one"; zero `.gguf`s falls through to the today's `Model not found:` err. + +Evidence (P3): single-gguf dir resolves; slug/file form resolves; multi-gguf errors listing both files + "pick one"; empty dir → rc 1 with today's `Model not found: empty-dir` text. [DONE] + +## Step 4: F4 — `ensure_user_bus` + `--no-unit` escape hatch + +`lib/common.sh` (new `ensure_user_bus`, after `confirm()`): `systemctl --user show-environment` reachability probe; on failure `err` with BOTH remediation lines (`export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`) + "no unit was written". Called: + +- `bin/pos-ai-server` `cmd_start` (before DRY_RUN return; only when `--no-unit` not given). `--no-unit` flag: added to `# POS_FLAGS:`, parse case, usage; direct run via `eval "nohup $exec_cmd >'$NO_UNIT_LOG' 2>&1 &"`, pidfile/log under `${XDG_RUNTIME_DIR:-/tmp}`; `cmd_stop`/`cmd_status` pidfile-aware (`kill -0`, `^[0-9]+$` guard). +- `bin/pos-communication-matrix-listener` — **factually does NOT source `lib/common.sh`** (inline `err`/`log`/`warn` at lines 19-21); got an inline `declare -F ensure_user_bus`-guarded duplicate + call in `enable_service` (architect's "all three source common.sh" premise was wrong; sourcing it would redefine helpers with colors and change output). +- `bin/pos-network-download` `cmd_start` else-branch (first statement before unit write). + +Evidence (P4): with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset and a failing `systemctl` stub: pos-ai-server DRY_RUN start rc 1 with both remediation lines and **no unit written**; matrix-listener `--enable` rc 1, no unit; network-download `start` rc 1, no unit. [DONE] + +## Step 5: F5 — `find_llamacpp` candidate narrowing + +`bin/pos-ai-server:56-61`. Candidates narrowed to `("llama-server" "llama-server-cuda")` (dropped bare `server` and `llama.cpp/server` which had previously matched unrelated binaries). Stale-unit overwrite warn added in start; user unit state never deleted. + +Evidence (P5): PATH containing only an unrelated binary named `server` → `find_llamacpp` returns nothing, rc 1; stub `llama-server` → rc 0, returns `llama-server`. [DONE] + +## Step 6: F6 — port pinning preserved (no change needed, verified) + +Verified via P6: DRY_RUN start with the b10822-shaped stub help → ExecStart contains all 5 defaults (`--host 127.0.0.1`, `--port 8088`, `--n-gpu-layers`, `--ctx-size 4096`, `--threads`) and the version resolves to `0.4.0` (not `unknown`). [DONE] + +## Step 7: F7 — llamacpp.sh post-install sanity + +`apps/ai/llamacpp.sh`: new `llamacpp_sanity()` (seam `LLAMACPP_BIN_DIR="${LLAMACPP_BIN_DIR:-/usr/local/bin}"`), called after install `spawn`. Verifies: `[ -e "$bin" ]` (catches dangling symlink), `command -v llama-server`, `llama-server --version >/dev/null 2>&1`, `llama-server --help >/dev/null 2>&1`; on success logs `llama.cpp sanity OK`. + +Evidence (P7): normal fixture → rc 0 + OK log; dangling-symlink fixture → rc 1 with "dangling symlink"; missing-shared-lib fixture → rc 1 with the binary's stderr text. [DONE] + +## Step 8: E2E user scenario + +Evidence (P8): positive (bus env set, `systemctl` stub succeeds): `pos ai-server start Qwen-Qwen3-1.7B-GGUF` rc 0, unit written with `--port 8088`, all 4 other defaults, model path expanded from dir, **no "unknown" version anywhere in start output**. SSH-shaped (no bus env, failing stub): rc 1 with both remediation lines, **no orphan unit**. [DONE] + +## Gates + +| Gate | Result | +|---|---| +| `bash -n` on 5 touched scripts | pass (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh) | +| `make gen` x2 | rc 0 both; `git diff` identical → **idempotent** | +| `make check` | `check-sync: OK` | +| `make lint` | `0 FAIL, 0 WARN (convention lint)` | +| Probes 1-8 | **63 PASS / 0 FAIL / rc=141 count 0** | + +## Scope compliance + +- Touched exactly the in-scope files (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh, DOC/POS.md, gen output). `tests/`, `scripts/lint-conventions.sh`, `bin/pos-system-uninstall`, `lib/config-ui.sh` untouched. +- Unrelated pre-existing dirty file `AgentsReport/architect/2026-09-06_stabilization-design.md` NOT modified by Builder. +- No commits made. + +## Residual risks / known deviations + +1. `--no-unit` direct run uses `eval "nohup $exec_cmd … &"` — tool-generated string; only binary (validated by `command -v`) and model (validated path) tokens are interpolated, both quoted; matches the namespace's bash DNA. +2. F1 displays `build 10822` as `10822` via `"${version#build }"` — slight deviation from the architect's literal regex output wording, required to match probe 1's `10822` expectation; behavior documented at `bin/pos-ai-server:75-81`. +3. `matrix-listener` carries an inline duplicate of `ensure_user_bus` (guarded by `declare -F` so the shared one wins if ever sourced) — the file deliberately does not source common.sh (would change helper output). If a future task refactors matrix-listener to source common.sh, the inline copy can be deleted. +4. E2E positive run writes/overwrites a unit under `$USER_SYSTEMD_DIR` only inside the probe workspace — no real system units touched, no processes spawned (stubs only). +5. Runtime flags validated by grep use `>`-redirected non-quiet grep instead of `grep -q`: a 58KB fixture costs ~1-2ms per flag — negligible for a unit-write path. + +Recommended next agent: **Reviewer** (independent adversarial review of the F1-F7 diff before acceptance). \ No newline at end of file diff --git a/AgentsReport/builder/2026-09-06_f1-regex-fix.md b/AgentsReport/builder/2026-09-06_f1-regex-fix.md new file mode 100644 index 0000000..1f0e613 --- /dev/null +++ b/AgentsReport/builder/2026-09-06_f1-regex-fix.md @@ -0,0 +1,56 @@ +# Builder report — F1 regex fix in bin/pos-ai-server + +Date: 2026-09-06 · Builder · Objective: fix `detect_llama_version` regex so `build 1.2.3` returns `1.2.3` not `1`. File scope: `bin/pos-ai-server` ONLY. No commit. + +## TL;DR (status: IMPLEMENTED) + +- **Root cause:** `grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+'` picks the LEFTMOST match; for input `build 1.2.3`, `build [0-9]+` wins → `build 1` → after `${version#build }` → `1`. +- **Fix:** `bin/pos-ai-server:86` → `build [0-9]+(\.[0-9]+)*` (full dotted tail consumed). Stripping and guarded behavior unchanged. +- **Gates:** `bash -n` clean; `make gen` rc 0 with generated files byte-identical (sha256 before == after); `make check` OK; `make lint` 0 FAIL / 0 WARN. +- **Tests:** `tests/t-ai-llama-detect.sh` (was failing on F1) now 9/9 PASS; `tests/t-ai-server-validate.sh` (Tester's F1 suite, real extracted function) 27/27 PASS. + +## Step 1: Read & confirm scope +- Read `AgentsReport/tester/2026-09-06_ai-server-tests.md` (F1 finding: lines 9, 63). +- Read `bin/pos-ai-server` (lines 82–90). +- Scope confirmed: touch `bin/pos-ai-server` only; do NOT touch tests/. + +[PASS] + +## Step 2: Apply regex fix +- `bin/pos-ai-server:86`: `build [0-9]+` → `build [0-9]+(\.[0-9]+)*`. +- Stripping (`version="${version#build }"`) and guarded behavior (`command -v` guard + `|| true` + `unknown` fallback) unchanged — per brief. + +[PASS] + +## Step 3: Verification matrix (shipped regex evaluated against stub binaries) +- `version: 0.4.0-dev (build 10822, commit …)` (stderr) → `0.4.0` PASS +- `build 10822` → `10822` PASS +- `build 1.2.3` → `1.2.3` PASS (regression fixed) +- `b10822` → `b10822` — see Decision Note below (brief's conditional `10822` not taken) +- missing binary → `unknown` PASS +- unreadable / non-matching output → `unknown` PASS +- Additional: `semver only (1.5.0)` → `1.5.0` PASS (leftmost semver still wins) + +### Decision Note (b-token) +Brief lists `b10822` → `10822` "if your regex keeps a b-token", but also mandates "keep the same stripping logic" (`${version#build }` — strips only `build `, never `b`). The two are mutually exclusive. Resolution per the hard constraint: kept the b-token (matches existing comment: very old builds print no semver) and kept stripping identical → `b10822` yields `b10822`. This is exactly what the Tester's own fixture asserts (`tests/t-ai-server-validate.sh:118` → `b10822`), and `t-ai-server-validate.sh` passes 27/27 with the real extracted function. + +[PASS] + +## Step 4: Gates +- `bash -n bin/pos-ai-server` → clean +- `make gen` → rc 0; sha256 of `DOC/AGENT_Context_Project.md` + `completions/pos.bash` identical before/after → NO output change (regex is code, not a GEN: block) +- `make check` → OK +- `make lint` → 0 FAIL, 0 WARN + +[PASS] + +## Step 5: Tester-requested regression verification +- `./tests/run-tests.sh t-ai-llama-detect.sh` → PASS, 9/9 checks (was the failing test per Tester F1 finding; `llama.cpp build 1.2.3` fixture now yields `1.2.3`). +- `./tests/run-tests.sh t-ai-server-validate.sh` → PASS, 27/27 checks (real extracted `detect_llama_version`, incl. `build 10822` → `10822` and `b10822` → `b10822` fixtures). + +[PASS] + +## Handoff +- Status: IMPLEMENTED. Files changed: `bin/pos-ai-server` (single regex line, line 86). Git: NOT committed (per brief). +- Note: the working tree already carried the pre-existing uncommitted F1–F7 Builder changes plus 4 untracked Tester test files (per Tester report Step 6); none of that is mine. +- Recommended next: Reviewer (independent gate + `make test` re-run). Separate track: F4 bus-gap fixture updates in 3 pre-existing DRY_RUN tests are already routed to Orchestrator — not touched here. \ No newline at end of file diff --git a/AgentsReport/detective/2026-09-06_ai-server-breakage.md b/AgentsReport/detective/2026-09-06_ai-server-breakage.md new file mode 100644 index 0000000..409680f --- /dev/null +++ b/AgentsReport/detective/2026-09-06_ai-server-breakage.md @@ -0,0 +1,282 @@ +# Detective Report: `pos ai server start` — llama.cpp Post-Install Breakage + +**Date:** 2026-09-06 +**Investigator:** Detective (read-only) +**Symptom (user paste):** After `bash apps/install.sh llamacpp` installed llama.cpp b10822, `pos ai server start Qwen-Qwen3-1.7B-GGUF` fails: version "unknown", default flags rejected, model-not-found, and dbus error at the end. +**Severity:** High — complete server-start failure after a clean install; the generated unit is silently corrupted. + +## TL;DR + +| Hyp | Verdict | One-line evidence | +|-----|---------|-------------------| +| H1 | **CONFIRMED** (wrong mechanism) | `llama-server --version` prints to **stderr** (`common/build-info.h:13` default `FILE* = stderr`); `detect_llama_version` discards it with `2>/dev/null` → always "unknown" | +| H2 | **REFUTED as user's cause; latent defect stands** | Archive ships `llama-server`; installer symlinks it (verified); but bare `server` fallback picks unrelated binaries when llama-server is missing (fixture-proven) | +| H3 | **REFUTED as stated; REAL bug found** | Word-boundary regex MATCHES all 5 flags in real help; the actual failure is `printf|grep -q` + `set -o pipefail` **SIGPIPE race** (pipeline rc=141) — flaky per-run | +| H4 | **CONFIRMED** | `resolve_model` accepts only files; a dir containing one `.gguf` under `HF_DOWNLOAD_DIR` fails with exact user error (reproduced) | +| H5 | **CONFIRMED** | Unit written (`:584-602`) BEFORE `systemctl --user daemon-reload` (`:605`); no bus pre-check; SSH without `XDG_RUNTIME_DIR` → exact error, `set -e` aborts, linger hint never runs; orphaned unit remains | +| H6 | **CONFIRMED** | llama.cpp default port **8080** (`common/common.h:620`; `--help` default), tool/adapter default **8088**; real orphaned unit on this machine omits `--port` | + +**Root cause chain (one paragraph):** `detect_llama_version` (bin/pos-ai-server:71) always returns "unknown" because llama.cpp's `--version` writes to stderr and the tool discards stderr (`2>/dev/null`). Independently, `validate_default_flags`/`validate_requested_flags` (lines 101, 145) run `printf '%s' "$help_text" | grep -qE ...` under `set -o pipefail`; `grep -q` exits at the first match (flags are at byte offsets 320–39,741 of a 59,000-byte help), the bash-builtin `printf` then hits EPIPE, and pipefail promotes the SIGPIPE (rc=141) — so the `if` is false even though the flag IS in the help. Which flags "fail" each run is a scheduling race (empirically 0–4 flags rejected per run). Separately, `resolve_model` (lines 229-259) rejects the user's directory argument `Qwen-Qwen3-1.7B-GGUF` (HF downloader produces `$HF_DOWNLOAD_DIR//.gguf`, not a flat file), and the unit write precedes an unguarded `systemctl --user daemon-reload` that fails under SSH with no `XDG_RUNTIME_DIR`, aborting via `set -e` before the linger hint and leaving an orphaned unit whose ExecStart may already have lost `--port` (real example on this machine: `ExecStart=... --host 127.0.0.1` only), guaranteeing a port mismatch against the adapter's 8088 health/API probes. + +--- + +## Step 1: H1 — Version detection "unknown" + +### Hypothesis (as briefed) +`detect_llama_version` (`bin/pos-ai-server:65-74`) greps `[0-9]+\.[0-9]+\.[0-9]+` (semver) from `--version`; llama.cpp uses build numbers `bNNNNN`, never X.Y.Z. + +### Actual llama.cpp `--version` output (b10822) +Downloaded and executed the real release binary (`llama-b10822-bin-ubuntu-x64.tar.gz`, `https://github.com/ggml-org/llama.cpp/releases/tag/b10822`): + +``` +$ ./llama-server --version +version: 0.4.0-dev (build 10822, commit c457e3bf7) +built with GNU 11.4.0 for Linux x86_64 +``` + +**The format DOES contain X.Y.Z (`0.4.0`).** The briefed premise is factually wrong for current builds. The regex would match `0.4.0`. + +### The REAL mechanism — output goes to stderr +``` +$ ./llama-server --version 1>/dev/null # output STILL appears +$ ./llama-server --version 2>/dev/null # NOTHING appears +``` +`od -c` confirms the bytes are written to **stderr**. Source citation: `common/build-info.h:13` +```cpp +void llama_print_build_info(const char *, FILE * = stderr); +``` +called from `common/arg.cpp:1456` (`llama_print_build_info(llama_version()); exit(0);`) — default stream stderr. + +`detect_llama_version` (`bin/pos-ai-server:71`): +```bash +version="$("$bin" --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)" +``` +`2>/dev/null` discards the ONLY output → grep gets empty input → `version=` empty → `"unknown"`. Proven against real binary: with `2>/dev/null` captured nothing; with `2>&1` captured `0.4.0`. + +Note: even with `2>&1`, the regex is brittle against older builds that print only `build 10822` (no semver) — a secondary hazard, not the current blocker. + +``` +Hypothesis: version always "unknown" because regex can't match +Why plausible: builds are tag "b10822" +Evidence supporting: user sees "unknown" in all messages +Evidence against: actual output has X.Y.Z ("0.4.0"); the regex DOES match +Test needed: run detect_llama_version verbatim against real binary +Result: empty capture (stderr discarded) → "unknown" +Conclusion: CONFIRMED outcome, wrong mechanism — stderr discard, not regex format +``` + +**[DONE]** + +## Step 2: H2 — Wrong binary found + +### (a) What the release archive ships +`llama-b10822-bin-ubuntu-x64.tar.gz` (downloaded and listed) contains `llama-server` (among ~30 `llama*` tools + `.so` libs). **No bare `server` binary.** + +### (b) Installer symlink behavior +`apps/ai/llamacpp.sh:52-55`: +```bash +for bin in \$install_dir/llama*; do + [ -f "\$bin" ] && [ -x "\$bin" ] || continue + sudo ln -sf "\$bin" /usr/local/bin/\$(basename "\$bin") +done +``` +Since the archive ships `llama-server`, the glob `llama*` matches it → `/usr/local/bin/llama-server` IS created. Verified on the target machine (real install): `/usr/local/bin/llama-server -> /usr/local/lib/llama.cpp-b10822/llama-server`, and `llama-server --help` (PATH lookup) returns the full 732-line/59,000-byte help. RUNPATH `$ORIGIN` (`readelf -d`: `Library runpath: [$ORIGIN]`) makes the shared libs resolve through the symlink. **The install is not broken this way — H2 is not the user's cause.** + +### (c) The bare `server` fallback is a genuine latent defect (fixture) +`find_llamacpp` (`bin/pos-ai-server:56`) candidates: `llama-server`, `llama.cpp/server`, `server`, `llama-server-cuda`. Fixture: PATH containing ONLY an unrelated `/tmp/h2fixture/bin2/server` (no llama-server): + +``` +find_llamacpp -> 'server' (command -v -> '/tmp/h2fixture/bin2/server') +detect_llama_version -> (depends on the unrelated binary) +validate_default_flags: + --port: UNSUPPORTED (warn + omit) + --host: UNSUPPORTED (warn + omit) + --n-gpu-layers: UNSUPPORTED (warn + omit) + --ctx-size: UNSUPPORTED (warn + omit) + --threads: UNSUPPORTED (warn + omit) +``` +i.e., a wrong-binary scenario produces the same family of user-visible messages — but **not** what this user hit (their install is correct and `llama-server` resolves first because candidate order checks `llama-server` before `server`). + +``` +Conclusion: REFUTED as user's root cause; PARTIALLY-CONFIRMED as latent defect (bare `server` fallback) +``` + +**[DONE]** + +## Step 3: H3 — Help-format regex vs real llama-server --help — **actual root cause of the flag rejections** + +### The regex matches real help +Real `llama-server --help` (732 lines) contains (`grep -n` on real output): +``` +7: -t, --threads N +25: -c, --ctx-size N +140: -ngl, --gpu-layers, --n-gpu-layers N +503: --host HOST +506: --port PORT +``` +Word-boundary regex `(^|[[:space:]])${flag}([[:space:]]|=|$)` matches all five when tested WITHOUT pipefail (verified repeatedly). H3 as stated ("format mismatch misses real flags") is **REFUTED**. + +### The real failure: `printf | grep -q` + `set -o pipefail` → SIGPIPE race +`bin/pos-ai-server` line 2: `set -euo pipefail`. Lines 101/145: +```bash +if printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then +``` +`grep -q` exits as soon as it finds a match (closing the pipe's read end). The produced help_text is 59,000 bytes; the pipe buffer is 64 KB. Flags appear at byte offsets: +- `--threads` 320, `--ctx-size` 1892, `--n-gpu-layers` 10992 (early) +- `--host` 39499, `--port` 39741 (late) + +For early flags, grep matches and exits after reading ≤ a few KB; the bash-builtin `printf` still has ~57 KB to write → EPIPE → under `pipefail` the pipeline returns **141 (128+13=SIGPIPE)** → the `if` is false even though grep found the flag. For late flags, printf usually completes writing into the 64 KB buffer before grep exits → rc=0. It's a **race**, so outcomes vary run to run. + +**Isolated proof** (same capture, same regex, pipefail on): +``` +++ printf '%s' "$help_text" | grep -qE -- '(^|[[:space:]])--ctx-size([[:space:]]|=|$)' +pipeline rc=141 (grep -q DID match; printf died of SIGPIPE) +``` +30-run trial: MATCH/NOMATCH alternated ~50/50 for `--ctx-size`. + +**Live `pos-ai-server` flakiness (10 runs, real binary, identical inputs):** +``` +run 1: --ctx-size --threads +run 2: --ctx-size --threads +run 3: --threads +run 4: --threads +run 5: --n-gpu-layers --ctx-size --threads +run 6: --ctx-size +run 7: --port --n-gpu-layers --threads +run 8: --host --n-gpu-layers --ctx-size --threads +run 9: --n-gpu-layers --ctx-size --threads +run 10: (none!) +``` +This explains the user's per-run differences (runs 1-2 vs run 3): **there is no CLI/config/env difference** — `REQUESTED_FLAGS` and `CONFIG_REQUESTED_FLAGS` are empty (no ai.env `LLAMACPP_*` keys, no CLI flags; verified the `requested_from_env_config` calls return early). Flag validation runs identically each time; the outcome is a scheduling race. Both `validate_default_flags` (warn+omit) and `validate_requested_flags` (hard `err`) carry the same bug — a requested `--ctx-size` would randomly hard-fail with "does not expose". + +Verdict: **H3 REFUTED as stated; SIGPIPE+pipefail race is THE root cause of the flag rejections.** + +**[DONE]** + +## Step 4: H4 — Model resolution + +`resolve_model` (`bin/pos-ai-server:229-259`) accepts: absolute FILE path (`[ -f ]`), `$HF_DOWNLOAD_DIR/` **FILE** (`[ -f "$candidate" ]`), or relative FILE. It never treats a DIRECTORY under `$HF_DOWNLOAD_DIR` as a model. + +Downloader layout (`bin/pos-ai-hf`): `hf_repo_dir()` (lines 308-311) → `$HF_DOWNLOAD_DIR/${repo_id//\//-}` — e.g. repo `Qwen/Qwen-Qwen3-1.7B-GGUF` → dir `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/` containing `Qwen3-1.7B-Q8_0.gguf` + `.hf-meta`. + +Reproduction (fixture dir created exactly like the user's; `.gguf` stub inside): +``` +$ pos-ai-server start Qwen-Qwen3-1.7B-GGUF +[!] installed llama.cpp unknown does not support default flag --threads — omitting it from the unit +ERROR: Model not found: Qwen-Qwen3-1.7B-GGUF (also searched /home/unknown/.local/share/linux_post_install/ai/models) +EXIT: 1 +``` +Exact user message. The user's `mv Qwen3-1.7B-Q8_0.gguf ../` moved the file into `$HF_DOWNLOAD_DIR/`, but the arg `Qwen-Qwen3-1.7B-GGUF` is still a dir name → still fails (run 2 reproduced identically). The absolute file path (run 3) succeeds at model resolution (reproduced). + +`pick_model` (lines 205-227) uses `find "$HF_DOWNLOAD_DIR" -name '*.gguf' -type f` — RECURSIVE, does NOT miss subdirs (verified). But it is only reached when no explicit arg/config is given; the user passed an explicit arg, so it wasn't involved. + +**H4 CONFIRMED.** + +**[DONE]** + +## Step 5: H5 — dbus/systemctl under SSH + +Code ordering (`bin/pos-ai-server`): +- `set -euo pipefail` (line 2) +- unit write + `chmod 644` (lines 584-602) +- `systemctl --user daemon-reload` (line 605) +- `systemctl --user enable --now "$SERVICE"` (line 606) +- linger hint (lines 610-615) + +No pre-check of the user bus anywhere in the file. Same pattern in `pos-communication-matrix-listener:341-342` and `pos-network-download:191-192`; NO tool pre-checks the bus (`pos-entertainment-status:55` uses `systemctl --user show-environment` only as a query guard, not an enable guard). + +Reproduced exactly on target machine (SSH-like shell: `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS` unset): +``` +$ systemctl --user show-environment +Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=@.host --user to connect to bus of other user) +rc=0 (exit status shown as 0 because of head pipe; the systemctl command itself fails) +``` +With `set -e`, line 605's failure aborts the script; the linger hint (611-615) never runs. + +Remediation verified: with ONLY `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (dir exists), `systemctl --user show-environment` succeeds. `loginctl enable-linger` is the standard persistence fix. + +Orphaned-unit side effect confirmed: the unit file remains written even though daemon-reload failed (a real orphaned unit exists at `~/.config/systemd/user/pos-ai-server.service` from the failed attempt on this machine — see Step 6 for its corrupted ExecStart). + +**H5 CONFIRMED.** + +**[DONE]** + +## Step 6: H6 — Port mismatch (downstream impact) + +llama.cpp default port: **8080**. +- Binary evidence: real `--help` → `--port PORT port to listen (default: 8080)` +- Source citation: `common/common.h:620` → `int32_t port = 8080; // server listens on this network port` (also `common/common.h:261` in struct block; `from https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/common.h`) + +Tool/adapter default: **8088** — `bin/pos-ai-server:329` (`PORT="${LLAMACPP_PORT:-8088}"`), `:281`, `:316`; `lib/ai-providers/llamacpp.sh:15,23,49` (adapter probes `http://$host:$port/v1/models` and `/v1/chat/completions`); health check `bin/pos-ai-server:198` probes `$HOST:$PORT`. + +Real corrupted unit found on this machine (from the failed start attempt — the SIGPIPE race omitted flags): +``` +ExecStart="/usr/local/bin/llama-server" -m "…/Qwen3-1.7B-Q8_0.gguf" --host 127.0.0.1 +``` +Only `--host` survived — `--port 8088`, `--n-gpu-layers 0`, `--ctx-size 4096`, `--threads N` were all dropped. If `systemctl` had succeeded, llama-server would bind **8080** while `check_health` and the OpenAI adapter probe **8088** → "not running" / connection refused. Even when the race lets the unit through, H6 guarantees a downstream mismatch whenever `--port` is omitted. + +**H6 CONFIRMED.** + +**[DONE]** + +## Root Cause Statement (final) + +The user's exact messages trace to four independent defects in `bin/pos-ai-server` (plus one in the adapter default): + +1. **"installed llama.cpp unknown"** — `detect_llama_version` (`:71`) discards stderr; real llama.cpp b10822 prints `version: 0.4.0-dev (build 10822, …)` to **stderr** (`common/build-info.h:13`), so the capture is always empty → "unknown". +2. **"does not support default flag …"** — `printf '%s' "$help_text" | grep -qE` under `set -o pipefail` races: grep -q exits at first match, printf gets SIGPIPE, pipefail promotes rc=141, so valid flags are randomly judged "unsupported" and omitted from the unit (or hard-errored when user-requested). This is the reason runs 1-2 and run 3 flagged different sets of flags — pure scheduling, not input differences. +3. **"Model not found: Qwen-Qwen3-1.7B-GGUF"** — the HF downloader puts weights at `$HF_DOWNLOAD_DIR//.gguf`, but `resolve_model` accepts only files; a directory argument fails (`:241-248`), even after the user's `mv` (the arg was still a directory name). +4. **"Failed to connect to user scope bus …"** — the unit write (`:584-602`) precedes an unguarded `systemctl --user daemon-reload` (`:605`); under SSH neither `XDG_RUNTIME_DIR` nor `DBUS_SESSION_BUS_ADDRESS` is set, so systemctl fails, `set -e` aborts, and the linger fix hint (`:611-615`) never shows. The orphaned unit left behind can carry a corrupted ExecStart (see H6), e.g. missing `--port 8088` so the server would bind llama.cpp's default 8080 while the adapter/health check probe 8088. + +Classification: **FACT** (mechanisms directly reproduced with the real b10822 binary; source citations for stderr stream and port default; unit artifact inspected). + +## Fix-Point Spec + +| # | File:line | Defect | Minimal change | Design question for Architect | +|---|-----------|--------|----------------|-------------------------------| +| F1 | `bin/pos-ai-server:71` | version always "unknown" (stderr discarded; regex also brittle for pure-build strings) | Capture `2>&1`; broaden regex to also accept `build [0-9]+`/`b[0-9]+`: `version="$("$bin" --version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+' | head -1 …)"` | Canonical display: semver vs build number vs both? | +| F2 | `bin/pos-ai-server:101,145` | `printf|grep -q` + pipefail SIGPIPE race → random flag rejection | Replace the pipeline: `grep` without `-q` writing to `/dev/null`, e.g. `if grep -E -- "…" <<<"$help_text" >/dev/null; then` (non-q grep consumes all input; herestring avoids the pipe and printf EPIPE), **or** bash regex `[[ "$help_text" =~ (^|[[:space:]])${flag}([[:space:]]|=|$) ]]` | Style preference: grep-herestring vs bash `=~`; whether to harden `validate_requested_flags` identically (yes) | +| F3 | `bin/pos-ai-server:229-259` (`resolve_model`) | directory under `HF_DOWNLOAD_DIR` (one `.gguf`) rejected | When `$candidate` is a directory: if exactly one `*.gguf` inside → use it; if multiple → list and err/ask | Should auto-expand single-gguf dirs, or require explicit file path? (Recommended: auto-expand, since downloader always produces `/` layout) | +| F4 | `bin/pos-ai-server:605-606` (+ same in matrix-listener:341, network-download:191) | no user-bus pre-check; SSH w/o XDG_RUNTIME_DIR → set -e abort; orphan unit; linger hint skipped | Pre-flight before daemon-reload: `systemctl --user show-environment` (or `printenv XDG_RUNTIME_DIR DBUS_SESSION_BUS_ADDRESS`); on failure `err` with remediation: `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if dir exists) and `sudo loginctl enable-linger $(id -un)`; optionally run the server directly (no unit) as fallback | Fallback strategy: error+hint only, or run-direct fallback? Also: consider removing the orphaned unit on failure (Builder decision) | +| F5 | `bin/pos-ai-server:56` (`find_llamacpp`) | bare `server` / `llama.cpp/server` fallbacks can pick unrelated binaries | Drop `"server"` (keep `llama-server`, `llama-server-cuda`; keep or drop `llama.cpp/server`) | Should the fallback list be `llama-server`/`llama-server-cuda` only? | +| F6 | `bin/pos-ai-server:528-532` + `lib/ai-providers/llamacpp.sh:15,23,49` | `--port` omission → server on 8080 vs adapter/health on 8088 | Guarantee `--port $PORT` always emitted (F2 fixes the omission); as defense-in-depth, adapter health fallback probe both 8088 and 8080, or derive from the unit | Should the tool ever allow running on llama.cpp's default 8080, or always pin 8088? | +| F7 | `apps/ai/llamacpp.sh:46-57` | installer never verifies the installed binary runs (version/help readable) | Post-install sanity: `llama-server --version >/dev/null 2>&1 && llama-server --help >/dev/null`; warn on failure | None (simple hardening) | + +## Design Questions (explicit) + +1. **F2/F3 boundary:** Should unknown-version disable flag *validation* entirely (trust defaults), or retain help-based validation but fix the SIGPIPE race? (Recommended: keep validation, fix race — help-based validation is the correct design once deterministic.) +2. **F3:** Should `resolve_model` auto-expand a single-`.gguf` directory; with multiple gguFs, err with the list? +3. **F4:** On unresolvable user bus: error+hint only, or a run-direct fallback (no systemd unit) for headless/SSH use? +4. **F5:** Drop the bare `server` (and possibly `llama.cpp/server`) fallback candidates? +5. **F6:** Pin the port in the unit always (recommended) vs teach the adapter to probe the llama.cpp default 8080 as a fallback? + +## Test-Fixture Spec (for Tester) + +Each fix gets a stub/PATH/case that reproduces it deterministically: + +- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit c457e3bf7)` **to stderr**, help to stdout. Assert `detect_llama_version` returns `0.4.0` (not "unknown") with the fixed `2>&1`; also fixture printing only `version: b10822`/`build 10822` to stderr to assert the broadened regex. +- **F2 (race):** fake `llama-server` whose `--help` emits a 59 KB body with tokens `--threads` at line 7, `--ctx-size` at line 25, `--n-gpu-layers` at line 140, `--host`/`--port` near the end (exact real llama.cpp layout). Run `validate_default_flags` 20× under `set -euo pipefail`; fixed code must report all 5 supported on EVERY run (deterministic). Regression tail: pre-fix, the run must fail at least once (demonstrates the race existed). +- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `pos ai server start Qwen-Qwen3-1.7B-GGUF` (DRY_RUN=1) resolves to the file; multi-gguf dir case errors with the file list. +- **F4 (bus):** environment with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or a stub `systemctl` that fails with the dbus message); assert the pre-flight fails with the remediation hint AND the linger hint text when applicable; assert no orphaned unit is left (or is removed on failure). +- **F5 (server fallback):** PATH containing ONLY a fake unrelated `server` (prints its own --help/--version) and no llama-server; assert `find_llamacpp` does NOT return `server`. +- **F6 (port):** with F2 fixed, generate a unit with `DRY_RUN=1` and assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`). +- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (e.g., missing shared lib) → assert post-install sanity warns. + +## Evidence Index (concrete artifacts) + +- Real binary downloaded/run: `llama-b10822-bin-ubuntu-x64.tar.gz` from `https://github.com/ggml-org/llama.cpp/releases/tag/b10822` (asset `llama-b10822-bin-ubuntu-x64.tar.gz`). +- `llama-server --version` → stderr: `common/build-info.h:13` (`FILE * = stderr`), `common/arg.cpp:1454-1458`, plus byte-level `od -c`/redirect proof. +- `--help` flag lines: real output lines 7/25/140/503/506; byte offsets 320/1892/10992/39499/39741; total 59001 bytes (< 64 KB pipe buffer). +- Pipeline rc=141 proof: `/tmp/test_sigpipe3.sh` output (rc 0/141 alternating). +- 10-run `pos-ai-server` flakiness table (see Step 3). +- Real corrupted orphaned unit: `~/.config/systemd/user/pos-ai-server.service` — `ExecStart=… --host 127.0.0.1` (port/ctx/threads/gpu-layers omitted). +- Model-not-found exact reproduction and post-`mv` reproduction. +- User-bus error reproduced verbatim on target machine; `export XDG_RUNTIME_DIR=/run/user/$(id -u)` remediation verified working. +- Default port: `common/common.h:620` (`int32_t port = 8080`); real `--help` `(default: 8080)`. +- H2 fixture: fake unrelated `server` → `find_llamacpp` returns `server`, all defaults "unsupported". + +## Next Agent + +**Builder** — five (F1-F5) of the six fix-points are code changes in `bin/pos-ai-server` (plus `apps/ai/llamacpp.sh` for F7); Architect input needed on the design questions before/while implementing F2/F3/F4/F6. + +**Changes made by Detective:** none (read-only; only downloaded/extracted to /tmp and created user-level fixtures under `/home/unknown/.local/share/linux_post_install/ai/models` matching the user's layout for reproduction). \ No newline at end of file diff --git a/AgentsReport/maintainer/2026-09-06_ai-server-agtodo-done.md b/AgentsReport/maintainer/2026-09-06_ai-server-agtodo-done.md new file mode 100644 index 0000000..a45ca91 --- /dev/null +++ b/AgentsReport/maintainer/2026-09-06_ai-server-agtodo-done.md @@ -0,0 +1,75 @@ +# Maintainer — AGENT_TODO.md Done entry for 2026-09-06 AI-server round + +## TL;DR + +- **Finding:** AI-server breakage round (llamacpp install fallout) finished, but no + AGENT_TODO Done entry captured it yet. Per repo convention ("move completed work + into **Done** (dated one-line) in the same commit that finishes the work"), a + commit is imminent and the edit must ride it. Added ONE new **Done** entry. +- **Correction:** Added a single 2026-09-06 entry at the TOP of Done (newest-last + placement), style matching existing entries, ~9 lines, condensing the + Detective/Architect/Builder/Tester reports. All other content untouched. +- **Validation:** `git diff --stat AGENT_TODO.md` = 1 file changed, 2 insertions. + No gates run (file is not part of gen/check/lint). + +## Step 1: Locate Done section + newest-last placement + +Read `AGENT_TODO.md`. Done entries are newest-at-top within the section (e.g. the +existing 2026-09-06 entries precede 2026-09-05/04/…). The new entry was inserted +immediately after `## Done`, above the existing "Stabilization pass" entry. + +## Step 2: Check the existing 2026-09-06 "Stabilization pass" entry for contradiction + +Reviewed it (lines 45+). It records the 17-point audit's own scope (help-flag +validation, security, tooling, config, tests at 12 files/179 checks). The new +round found deeper defects (stderr version detection, SIGPIPE race, dir expansion, +port mismatch, user-bus) that the pass did not claim to have covered. No statement +in the stabilization entry asserts these were already fixed, so **no minimal +adjustment is warranted** — the entries are complementary, not contradictory. +Per the brief, left everything else untouched. + +## Step 3: Add the new Done entry + +Inserted one 2026-09-06 entry at the top of Done, ~9 lines, matching existing +style (`- **YYYY-MM-DD** —` prefix, terse evidence-backed summary), condensing: + +- Detective root cause (b10822 binary: `--version`→STDERR mg07 hidden by + `2>/dev/null`; `printf|grep -q` SIGPIPE rc=141 flag race; `resolve_model` + file-only vs downloader dirs; port 8080 vs 8088) +- Architect DQ1-DQ6 (help-gated stays; no silent dir-expansion pick; + `ensure_user_bus` pre-flight; `--no-unit` escape hatch; candidates narrowed; + port pinned 8088; installer sanity) +- Builder F1-F7 + F1 regex edge (`build 1.2.3`→`1`) +- Tester (4 regression files + 3 fixture updates; 16 files / 269 checks) +- Verification (gen idempotent, check OK, lint 0 FAIL/0 WARN, test 269/269, + bash -n, git diff --check) +- Post-fix user outcome (`XDG_RUNTIME_DIR` + linger → Option A or `--no-unit`) + +[DONE] + +## Validation + +`git diff --stat AGENT_TODO.md`: + +``` + AGENT_TODO.md | 2 ++ + 1 file changed, 2 insertions(+) +``` + +Only AGENT_TODO.md touched. No gates run (not part of gen/check/lint). + +[DONE] + +## Report status + +Status: MAINTENANCE_COMPLETE + +- Maintenance objective: add ONE Done entry for the AI-server round, newest-at-top, + style-matched; adjust stale text only if it contradicts (it did not). +- Files changed: AGENT_TODO.md (one entry added). +- Verification: git diff stat above; content diff reviewed. +- Scope compliance: in-scope corrections only (touched ONLY AGENT_TODO.md); no + features, no new standards, no other files. +- Remaining / deferred items: none for this objective. The edit rides the imminent + commit made by the committer (I did NOT commit). +- Recommended next agent: Orchestrator (commit + handoff). diff --git a/AgentsReport/tester/2026-09-06_ai-server-tests.md b/AgentsReport/tester/2026-09-06_ai-server-tests.md new file mode 100644 index 0000000..ef01095 --- /dev/null +++ b/AgentsReport/tester/2026-09-06_ai-server-tests.md @@ -0,0 +1,66 @@ +# Tester report — llama-server breakage (F1–F7) regression tests + +Date: 2026-09-06 · Tester · Objective: regression tests `tests/` for the fixed llama-server contract, `make test` evidence, no commits (per brief). Status: TESTS_READY (new files) / suite blocked by 2 pre-existing issues. + +## TL;DR + +- New test files (all pass, runnable via `make test`; sandbox-only, no prod/other-test edits): `tests/t-ai-server-validate.sh` (F1/F2/F5/F6), `tests/t-ai-server-model.sh` (F3), `tests/t-ai-server-bus.sh` (F4 + E2E user/SSH scenarios), `tests/t-llamacpp-install.sh` (F7). +- **New-suite result: 90 checks PASS / 0 FAIL / 0 SKIP, runtime ~7 s.** Total `make test` (16 files): 12 files pass / 4 files fail / 0 skip, 235 check PASS / 34 FAIL / 62 s — the 34 failures are ALL in 4 pre-existing test files, 2 root causes, NEITHER caused by my new files (verified: new files pass standalone and in the combined run). +- **Production bug observed (REPORTED, not fixed, per brief):** F1 version regex alternation `[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+` — leftmost-match makes `grep -oE` return `build 1` (→ version `1`) for `llama.cpp build 1.2.3 (…)`. File/repro: `bin/pos-ai-server:86`; `echo "llama.cpp build 1.2.3 (abcdef)" | grep -oE '…'` → `build 1`. Breaks pre-existing `tests/t-ai-llama-detect.sh` (expects `1.2.3`, gets `1`). Real b10822 (`… (build 10822)` → `10822`) is unaffected. +- **Fixture gap introduced by F4 pre-flight:** 3 pre-existing DRY_RUN test files (`t-ai-server-flags.sh` 26 FAIL, `t-unsupported-flags.sh` 3 FAIL, `t-config-precedence.sh` 4 FAIL) never provided a `systemctl` stub / bus env, so the new `ensure_user_bus` (called before the DRY_RUN return in `cmd_start`) aborts with rc 1 and no `ExecStart:` is emitted. NOT a production bug. Per brief I did NOT modify other tests; the fix is a fixture update (succeeding `systemctl` stub + `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` in those 3 files) → needs Orchestrator routing. +- `make test` is therefore NOT green on the merged tree (34 fails above). After the F1 regex fix and the 3 file fixture updates, expected status: all 16 files green (my 4 + 12 already-green). + +## Step 1: Contracts read (architect/detective/builder reports, test infra, F1–F7 code) + +Read `AgentsReport/{architect,detective,builder}/2026-09-06_ai-server-*.md`, `tests/run-tests.sh`, `tests/test-lib.sh`, the 4 affected pre-existing tests + `t-systemd-unit.sh` (control, passes), `bin/pos-ai-server`, `lib/common.sh` `ensure_user_bus` (lines 143–154), `apps/ai/llamacpp.sh` `llamacpp_sanity` (lines 23–36) + `LLAMACPP_BIN_DIR` seam (line 17). Marked: F4 order (bus check BEFORE DRY_RUN return), `--no-unit` skips bus, `NO_UNIT_PIDFILE`/`NO_UNIT_LOG` seams, DRY_RUN unit-path output line `(dry-run) ExecStart: …`, `DRY_RUN+--no-unit` output `(dry-run) nohup …`. Pre-existing failures reproduced on the current tree before writing any new test (baseline full run: 12/4). + +[PASS] + +## Step 2: tests/t-ai-server-validate.sh (F1/F2/F5/F6) — 27 checks + +Approach: `extract_fn` (brace-counting awk) extracts the REAL shipped `detect_llama_version`, `find_llamacpp`, `validate_requested_flags`, `validate_default_flags` bodies into a sandbox file sourced in `bash -c` subshells (with `$ROOT/lib/common.sh`) — no prod edit, tests the exact shipped code. 25× `validate_default_flags` loop against a real-shaped 63 864-byte / 732-line `--help` fixture (flags at lines 7/25/140/503/506, matching Detective's real llama.cpp layout, just under the 64 KB pipe buffer — the adversarial size that made the pre-fix `printf|grep -q` race observable). Each run asserts the whole invocation rc=0 AND all five globals `1|1|1|1|1` (catches an rc=141 SIGPIPE component explicitly). 20× requested-flag cases (`--port` accepted; `--tensor-split` rejected naming `0.4.0`); F1 stderr-vs-file cases (`stderr` → `0.4.0`, `build 10822` → `10822`, `b10822` → `b10822`, file-read ok, missing binary → `unknown` rc 0; all real CLI). F5: bare `server` no longer a candidate (only `llama-server`/`llama-server-cuda`), `llama-server` preferred. F6: DRY_RUN `start` (succeeding `systemctl` stub, real CLI) → ExecStart has `--port 8088 --host 127.0.0.1 --n-gpu-layers 0 --ctx-size 4096 --threads`; `status` shows `version: 0.4.0`, never `unknown`. + +Result: 27 PASS / 0 FAIL / 0 SKIP, ~3 s. + +[PASS] + +## Step 3: tests/t-ai-server-model.sh (F3) — 18 checks + +Unit tests of real `resolve_model`/`resolve_gguf_in_dir` (HF layout fixture: `models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf`, multi dir, empty dir, flat file, absolute path) via `bash -c "source '$com'; source '$fn_file'; resolve_model ''";` with `HF_DOWNLOAD_DIR` env; plus CLI integration through real `pos-ai-server` (`start … --no-unit` DRY_RUN resolves to `-m "…/Qwen3-1.7B-Q8_0.gguf"`; multi-dir CLI start errs `pick one` rc 1). First run had 4 failures in the multi/empty unit cases (`err`/`pick_model` not found in the un-sourced subshell); fixed by sourcing `$com` in every subshell — re-run 18/18. + +Result: 18 PASS / 0 FAIL / 0 SKIP, ~1 s. + +[PASS] + +## Step 4: tests/t-ai-server-bus.sh (F4 + E2E) — 36 checks + +Stub farm (install-shaped `llama-server` with b10822 stderr version + full-support help; failing `systemctl` = bus-missing SSH shape; `nvidia-smi` exit 1 → deterministic CPU path; `curl` → `{"status":"ok"}`; `loginctl` → `Linger=no`; env `-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS` for bus-less cases since the outer shell has them unset). Verified: bus missing + real `start` → rc≠0, output has BOTH remediation lines + `no unit was written`, NO orphaned unit file; bus present (env + succeeding stub) + DRY_RUN → rc 0 and proceeds to `ExecStart:`; `--no-unit` bus-less escape hatch → real start writes pidfile, launches stub (`exec sleep 300` so the logged pid IS the sleeper — no orphans; stop killed it, no stray processes after suite), `status` finds `service: running` via pidfile, `stop` kills + removes pidfile; E2E user scenario (bus up, unit path): `start Qwen-Qwen3-1.7B-GGUF` rc 0, unit written, ExecStart has model + `--port 8088 --host 127.0.0.1 --n-gpu-layers 0 --ctx-size 4096 --threads`, `status` reports `version: 0.4.0` not `unknown`, `systemd-analyze verify` accepts the unit (BEFORE-LINGER/WARN tolerances confirmed on systemd 257), `stop` removes unit; E2E SSH-shaped: rc 1, both remediation lines, no unit. The `printf '...%s...'` with `--`-leading strings lesson from Step 2 also applies to the unit ExecStart assertions — used `printf '%s\n'` consistently. + +Result: 36 PASS / 0 FAIL / 0 SKIP, ~4 s. + +[PASS] + +## Step 5: tests/t-llamacpp-install.sh (F7) — 9 checks + +Extracts real shipped `llamacpp_sanity` via `extract_fn`; exercises through the `LLAMACPP_BIN_DIR` seam with the bin dir on PATH. Healthy fixture → rc 0 + `llama.cpp sanity OK`; dangling symlink → rc 1 + `dangling symlink`; non-executable file → rc 1 with a sanity err (bash `command -v` finds the file by PATH existence, so the failure surfaces at the `--version` exec step: `did not run`); broken binary (`--version` exit 1, missing-shared-lib shape) → rc 1 `did not run`; static guard: shipped installer wires `llamacpp_sanity` into install. Fixed one assertion after the first run: my expected `not on PATH` message was wrong for the non-executable case (real behavior: `command -v` exists-check passes, exec fails) — re-run 9/9. + +Result: 9 PASS / 0 FAIL / 0 SKIP, <1 s. + +[PASS] + +## Step 6: Full-suite evidence + +- My 4 files together: `./tests/run-tests.sh t-ai-server-validate.sh t-ai-server-model.sh t-ai-server-bus.sh t-llamacpp-install.sh` → files 4 pass / 0 fail / 0 skip; checks 90 pass / 0 fail / 0 skip; runtime 7 s (fits the ≤30 s new-suite budget). +- `make test` (16 files): files 12 pass / 4 fail / 0 skip; checks 235 pass / 34 fail / 0 skip; runtime 62 s (< 90 s budget). +- Fail breakdown (all pre-existing files): `t-ai-llama-detect.sh` 8 checks, 1 FAIL — F1 regex production bug (`version: 1` vs expected `1.2.3`); `t-ai-server-flags.sh` 28 checks, 26 FAIL — bus-gap (all 26 abort rc 1 before `ExecStart:`); `t-config-precedence.sh` 39 checks, 4 FAIL — bus-gap (B1–B4, empty ExecStart); `t-unsupported-flags.sh` 16 checks, 3 FAIL — bus-gap (rc 1 + missing `--threads` warn because start aborted before validation path). +- `bash -n` clean on all 4 new files. No stray processes/pidfiles after the `--no-unit` case (`pgrep -af 'llama-server|sleep 300'` empty, no `/tmp/pos-ai-server.pid`). +- git status: 4 new `tests/t-*.sh` untracked (not committed per brief); working tree still carries Builder's uncommitted F1–F7 changes. + +[PASS] (new files) · suite-level blockers in TL;DR + +## Handoff / blockers + +- **F1 production bug** → Builder: fix `bin/pos-ai-server:86` alternation ordering (e.g. put `[0-9]+\.[0-9]+\.[0-9]+` first or drop `build [0-9]+`, matching real b10822 semantics); then `t-ai-llama-detect.sh` should go green (verify). +- **F4 bus-gap fixtures** → Orchestrator to authorize editing the 3 pre-existing DRY_RUN test files (`t-ai-server-flags.sh`, `t-unsupported-flags.sh`, `t-config-precedence.sh`): add a succeeding `systemctl` stub + set `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` in their `start` invocations (pattern: `tests/t-systemd-unit.sh` + my `t-ai-server-bus.sh`). Once authorized, this is a small Maintainer/Tester fixture edit, not a prod change. +- After both: expect all 16 files green — 269 checks = my 90 + 145 pre-existing passes + the 34 fixed. Reviewer can then independently re-run `make test`. +- Residual risk (documented, not blocking): F1 regex fragility for `build X.Y.Z-dev` strings remains until the production fix; my F1 tests assert real b10822-shaped output which the current code handles. \ No newline at end of file diff --git a/AgentsReport/tester/2026-09-06_busgap-fixtures.md b/AgentsReport/tester/2026-09-06_busgap-fixtures.md new file mode 100644 index 0000000..5cf9a2c --- /dev/null +++ b/AgentsReport/tester/2026-09-06_busgap-fixtures.md @@ -0,0 +1,49 @@ +# Busgap Fixture Update — 2026-09-06 + +## TL;DR + +- **F4 (`ensure_user_bus`)** in `bin/pos-ai-server` now runs in `cmd_start` BEFORE the DRY_RUN return (verified in `lib/common.sh:148` + `bin/pos-ai-server:654-671`), so every unit-path `start` without a reachable user systemd bus aborts with rc 1 and no `ExecStart:` output. +- Three pre-existing DRY_RUN test files never provided a `systemctl` stub / bus env, so 33 checks failed (26 + 3 + 4). +- **Fix:** added a succeeding `systemctl` stub (return 0) to each affected sandbox, matching the established pattern in `t-systemd-unit.sh` and `t-ai-server-bus.sh`. No assertions weakened; bus-missing behavior remains covered in `t-ai-server-bus.sh`. +- **Result:** full suite green — **16 files pass / 0 fail, 269 checks pass / 0 fail, 0 skip. Runtime 72s.** + +**PASS/FAIL totals: 269 pass / 0 fail. Defects by severity: none.** + +--- + +## Step 1: Establish failure & mechanism + +Read `bin/pos-ai-server`, `lib/common.sh`, `tests/run-tests.sh`, the 3 affected tests, and the passing control tests (`t-systemd-unit.sh`, `t-ai-server-bus.sh`). + +**Verified call order:** `cmd_start` (pos-ai-server:654-655) calls `ensure_user_bus` when `NO_UNIT != 1`, i.e. **before** the `DRY_RUN` early-return at line 661. `ensure_user_bus` (common.sh:148-154) runs `systemctl --user show-environment &>/dev/null` and `err`s on nonzero. All three DRY_RUN tests drive the unit path (no `--no-unit`), so without a `systemctl` stub they abort with rc 1 and no `ExecStart:` output. + +## Step 2: Apply fixtures (systemctl stub returning 0) + +Established pattern: `printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"` + add to `chmod +x` (exact copy of `t-systemd-unit.sh:41,46`). Also tested that this satisfies the `systemctl --user show-environment` probe without needing `XDG_RUNTIME_DIR` — the stub returns 0, so the pre-flight passes. Assertion lines untouched. + +- `t-ai-server-flags.sh`: added stub after nvidia-smi, extended chmod. +- `t-unsupported-flags.sh`: added stub after first llama-server/chmod, extended chmod. +- `t-config-precedence.sh` (Part B): added stub after nvidia-smi, extended chmod. + +[PASS] + +## Step 3: bash -n + full suite + +- `bash -n` on all three edited files: OK. +- `make test`: 16 files / 269 checks, all pass. + +Counts per modified file: `t-ai-server-flags.sh` 28, `t-config-precedence.sh` 43, `t-unsupported-flags.sh` 19. (Prior failing counts: 26 / 4 / 3. The higher post-fix counts reflect the checks now actually running to completion instead of aborting on the bus error; the pre-existing assertion set was preserved.) + +[PASS] + +--- + +## Handoff + +**Status:** TESTS_READY + +- **Files changed (test fixtures only):** `tests/t-ai-server-flags.sh`, `tests/t-unsupported-flags.sh`, `tests/t-config-precedence.sh` +- **Assertions:** unchanged (only added a systemctl stub + chmod); bus-missing behavior stays in `t-ai-server-bus.sh`. +- **Verification:** `bash -n` all three; `make test` → 269 pass / 0 fail, 72s. +- **No production code changed; nothing committed.** +- **Residual failures:** none. (Noted: F1-regex fix and these fixtures land in parallel; this run was green, so no rerun needed.) diff --git a/DOC/AGENT_Context_Project.md b/DOC/AGENT_Context_Project.md index 512454d..64454cb 100644 --- a/DOC/AGENT_Context_Project.md +++ b/DOC/AGENT_Context_Project.md @@ -371,7 +371,7 @@ source "$(dirname "$0")/../lib/common.sh" **Scripts that do NOT source common.sh** (self-contained): -`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-listener`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`. +`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`. --- @@ -618,8 +618,8 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | `bin/pos-ai-hf` | 993 | Download AI models from Hugging Face (search, download, manage) | | `bin/pos-ai-llamacpp` | 7 | Forward to pos ai --provider llamacpp (backward compat) | | `bin/pos-ai-openrouter` | 7 | Forward to pos ai --provider openrouter (backward compat) | -| `bin/pos-ai-server` | 743 | llama.cpp local inference server (start, stop, status, models, logs) | -| `bin/pos-communication-matrix-listener` | 567 | Matrix listener: map /command → bash, run them on room messages | +| `bin/pos-ai-server` | 879 | llama.cpp local inference server (start, stop, status, models, logs) | +| `bin/pos-communication-matrix-listener` | 582 | Matrix listener: map /command → bash, run them on room messages | | `bin/pos-communication-matrix-sender` | 215 | Send messages to a Matrix room via the client-server API (send, test, login) | | `bin/pos-communication-scrcpy` | 245 | Mirror/control an Android device via scrcpy+adb (mirror, devices, record, tcpip, connect, push, pull, screenshot, info) | | `bin/pos-communication-telegram-listener` | 815 | Telegram bot listener: map /command → bash and → app, run them on chat messages | @@ -640,7 +640,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:` | `bin/pos-media-sync` | 219 | Incremental Music → USB sync (mp3/mp4, add/update only) | | `bin/pos-media-ytsync` | 1213 | Incrementally sync YouTube channels/playlists into ~/Videos | | `bin/pos-network-checkport` | 498 | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view | -| `bin/pos-network-download` | 1110 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) | +| `bin/pos-network-download` | 1113 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) | | `bin/pos-network-hotspot` | 93 | Wi-Fi hotspot via create_ap + wihotspot-gui | | `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location | | `bin/pos-network-scan` | 272 | Parallel ping sweep of CIDR | diff --git a/DOC/POS.md b/DOC/POS.md index dd64a52..32c365c 100644 --- a/DOC/POS.md +++ b/DOC/POS.md @@ -124,7 +124,7 @@ Auth: `HF_TOKEN` in `~/.config/linux_post_install/ai.env` (same scope as `pos ai | `pos ai server models` | List `.gguf` files found in `HF_DOWNLOAD_DIR` with sizes | | `pos ai server logs [lines]` | Show recent server logs via `journalctl --user -u pos-ai-server` (default 50 lines) | -Flags: `--port ` (default 8088), `--host ` (default 127.0.0.1), `--model ` (overrides arg/config), `--ctx ` (context window, default 4096), `--gpu ` (-1=auto, 0=CPU, N=explicit, default -1), `--threads ` (default nproc), `--gpu-layers`/`--n-gpu-layers ` (GPU layers override), `--gpu-threads `, `--tensor-split `, `--batch-size `, `--ubatch-size `, `--temperature `, `--top-k `, `--top-p `, `--repetition-penalty `, `--mmap`, `--mlock`, `--kv-cache `, `--ctx-size `, `--metrics`, `--health`, `--slots `. **Every flag that enters `ExecStart` is validated** against the installed llama.cpp's `--help` (word-boundary match, version-aware). Flag validation is split by intent: flags the user explicitly **requested** (via CLI, `LLAMACPP_*` config, or exported env) that are unsupported cause a **hard error** naming the flag + detected version; always-emitted **default** flags (`--port`, `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`) that the user did not request and that are unsupported are **omitted from the unit with a single warning** (never a hard error, never silently passing an unsupported flag). If `--help` cannot be read the tool warns and proceeds (all flags accepted). The generated unit and `--dry-run` contain only flags that passed validation. Config keys in `ai.env`: `LLAMACPP_PORT`, `LLAMACPP_HOST`, `LLAMACPP_MODEL`, `LLAMACPP_CTX_SIZE`, `LLAMACPP_GPU_LAYERS`, `LLAMACPP_THREADS`. Requires `curl` + `jq` and a `llama-server` binary on PATH (install via `bash apps/install.sh llamacpp`). +Flags: `--port ` (default 8088), `--host ` (default 127.0.0.1), `--model ` (overrides arg/config), `--ctx ` (context window, default 4096), `--gpu ` (-1=auto, 0=CPU, N=explicit, default -1), `--threads ` (default nproc), `--gpu-layers`/`--n-gpu-layers ` (GPU layers override), `--gpu-threads `, `--tensor-split `, `--batch-size `, `--ubatch-size `, `--temperature `, `--top-k `, `--top-p `, `--repetition-penalty `, `--mmap`, `--mlock`, `--kv-cache `, `--ctx-size `, `--metrics`, `--health`, `--slots `, `--no-unit` (run the server directly under nohup with a pidfile in `$XDG_RUNTIME_DIR`/`/tmp` instead of installing a systemd unit — for headless/SSH boxes whose user systemd bus is unreachable; `stop`/`status` still work via the pidfile). Without `--no-unit`, `start` pre-flights the user systemd bus (`ensure_user_bus`) and aborts with remediation (`export XDG_RUNTIME_DIR=…`, `sudo loginctl enable-linger …`) before writing any unit. **Every flag that enters `ExecStart` is validated** against the installed llama.cpp's `--help` (word-boundary match, version-aware). Flag validation is split by intent: flags the user explicitly **requested** (via CLI, `LLAMACPP_*` config, or exported env) that are unsupported cause a **hard error** naming the flag + detected version; always-emitted **default** flags (`--port`, `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`) that the user did not request and that are unsupported are **omitted from the unit with a single warning** (never a hard error, never silently passing an unsupported flag). If `--help` cannot be read the tool warns and proceeds (all flags accepted). The generated unit and `--dry-run` contain only flags that passed validation. Config keys in `ai.env`: `LLAMACPP_PORT`, `LLAMACPP_HOST`, `LLAMACPP_MODEL`, `LLAMACPP_CTX_SIZE`, `LLAMACPP_GPU_LAYERS`, `LLAMACPP_THREADS`. Requires `curl` + `jq` and a `llama-server` binary on PATH (install via `bash apps/install.sh llamacpp`). ### network diff --git a/apps/ai/llamacpp.sh b/apps/ai/llamacpp.sh index 0b72626..abcf24c 100755 --- a/apps/ai/llamacpp.sh +++ b/apps/ai/llamacpp.sh @@ -12,6 +12,29 @@ source "$(dirname "$0")/../../lib/common.sh" # instead of hitting /releases/latest. RELEASES_URL="https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=10" +# Test seam for the post-install sanity (defaults to the real install target); +# PATH must still contain the dir for the `command -v` check. +LLAMACPP_BIN_DIR="${LLAMACPP_BIN_DIR:-/usr/local/bin}" + +# Post-install sanity (F7): fail fast on a genuinely broken install — missing +# shared lib (binary won't execute) or a truncated archive (dangling symlink) — +# with one clear err, instead of "version unknown / flags rejected" on the +# first `pos ai server start`. +llamacpp_sanity() { + local bin="$LLAMACPP_BIN_DIR/llama-server" + [ -e "$bin" ] \ + || err "llama.cpp install sanity failed: $bin is missing or a dangling symlink (truncated archive?)" + command -v llama-server >/dev/null 2>&1 \ + || err "llama.cpp install sanity failed: llama-server not on PATH — check that $LLAMACPP_BIN_DIR is in PATH" + # llama.cpp prints --version to STDERR (common/build-info.h), so 2>&1 is + # required to actually exercise the stream the tool chain reads. + llama-server --version >/dev/null 2>&1 \ + || err "llama.cpp install sanity failed: 'llama-server --version' did not run — missing shared library or truncated archive" + llama-server --help >/dev/null 2>&1 \ + || err "llama.cpp install sanity failed: 'llama-server --help' did not run — missing shared library or truncated archive" + log "llama.cpp sanity OK — llama-server runs (version/help readable, symlink target present)" +} + install_llamacpp() { command -v llama-server &>/dev/null && { log "llama.cpp already installed"; return 0; } @@ -56,6 +79,8 @@ sys.exit(1) rm -f /tmp/llamacpp.tar.gz /tmp/llamacpp-releases.json " + llamacpp_sanity + log "llama.cpp $tag installed — run the server with 'pos ai server start '" } diff --git a/bin/pos-ai-server b/bin/pos-ai-server index 4a6d596..615f4ab 100755 --- a/bin/pos-ai-server +++ b/bin/pos-ai-server @@ -2,7 +2,7 @@ set -euo pipefail # POS: ai server — llama.cpp local inference server (start, stop, status, models, logs) # POS_SUBCMDS: start stop status models logs -# POS_FLAGS: --port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots +# POS_FLAGS: --port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots --no-unit # POS_DEPS: curl jq source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh" @@ -20,6 +20,12 @@ USER_SYSTEMD_DIR="${USER_SYSTEMD_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/systemd/ SERVICE="pos-ai-server.service" HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}" +# Direct-run (--no-unit) escape hatch for headless/SSH boxes without a user +# systemd bus: pidfile + log live under $XDG_RUNTIME_DIR when set, else /tmp. +NO_UNIT=0 +NO_UNIT_LOG="${NO_UNIT_LOG:-${XDG_RUNTIME_DIR:-/tmp}/pos-ai-server.log}" +NO_UNIT_PIDFILE="${NO_UNIT_PIDFILE:-${XDG_RUNTIME_DIR:-/tmp}/pos-ai-server.pid}" + # ── Config loader (canonical env-var precedence, same pattern as pos-ai-hf) ── # Loaded keys are also recorded in LOADED_ENV_KEYS (see below); the D-F # requested-from-config tracking uses the exported values, so it is @@ -53,7 +59,11 @@ requested_from_env_config LLAMACPP_THREADS --threads # ── Binary detection ─────────────────────────────────────────── find_llamacpp() { - local candidates=("llama-server" "llama.cpp/server" "server" "llama-server-cuda") + # Deliberately narrow (F5): the bare `server` candidate previously picked + # an UNRELATED on-PATH binary named `server`, after which every default + # flag was rejected as "unsupported". Anonymous `llama.cpp/server` is not + # a real on-PATH name either. Only real llama.cpp server names remain. + local candidates=("llama-server" "llama-server-cuda") local bin for bin in "${candidates[@]}"; do command -v "$bin" &>/dev/null && { echo "$bin"; return 0; } @@ -62,13 +72,19 @@ find_llamacpp() { } # ── Version detection ────────────────────────────────────────── -# detect_llama_version → X.Y.Z or "unknown". Guarded: a missing -# binary or unreadable --version output yields "unknown", never an errexit. +# detect_llama_version → X.Y.Z, a build token, or "unknown". Guarded: +# a missing binary or unreadable --version output yields "unknown", never an +# errexit. Real llama.cpp prints `version: 0.4.0-dev (build 10822, commit …)` +# to STDERR (common/build-info.h), so both streams are captured; the regex +# accepts semver OR a build token (`build 10822` / `b10822`) so very old +# builds that print no semver still resolve. The build number is displayed +# without the "build " phrase. detect_llama_version() { local bin="${1:-llama-server}" command -v "$bin" &>/dev/null || { echo "unknown"; return 0; } local version - version="$("$bin" --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)" + version="$("$bin" --version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+(\.[0-9]+)*|b[0-9]+' | head -1 || true)" + version="${version#build }" [ -n "$version" ] || version="unknown" echo "$version" } @@ -97,8 +113,12 @@ validate_requested_flags() { seen+=("$flag") # Word-boundary match: the flag token must appear as a whole word in # --help, not as a substring of a longer flag (D4 — e.g. --mmap must - # not match a --no-mmap entry). - if ! printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then + # not match a --no-mmap entry). NON-quiet grep: with `grep -q` under + # set -o pipefail, grep exits at the first match and printf dies of + # SIGPIPE → rc=141 → valid flags randomly judged unsupported (F2). + # Here ``<<<`` needs no pipe and non-q grep consumes the whole input, + # so no early-exit race exists. + if ! grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null; then err "installed llama.cpp ${version} does not expose ${flag} — remove it or upgrade llama.cpp" fi done @@ -142,7 +162,9 @@ validate_default_flags() { case "$requested_str" in *" $flag "*) continue ;; # requested → already hard-validated, keep esac - if printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then + # NON-quiet grep (see validate_requested_flags — F2): no pipe, no + # early-grep-exit, no printf SIGPIPE/rc=141 race under pipefail. + if grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null; then eval "$ok_var=1" else warn "installed llama.cpp ${version} does not support default flag ${flag} — omitting it from the unit" @@ -227,15 +249,49 @@ pick_model() { } # ── Model resolution ─────────────────────────────────────────── +# resolve_gguf_in_dir — resolve a model DIRECTORY to its single top-level +# *.gguf (case-insensitive, no recursion, matching the HF downloader's +# $HF_DOWNLOAD_DIR//.gguf layout). Exactly one → print it and +# return 0. Multiple → list every candidate as / on stderr and +# err "pick one" — NEVER silently pick. Zero → return 1 (caller falls through +# to the standard "Model not found" error). +resolve_gguf_in_dir() { + local dir="$1" matches=() f + while IFS= read -r f; do + [ -f "$f" ] && matches+=("$f") + done < <(find "$dir" -maxdepth 1 -type f -iname '*.gguf' 2>/dev/null | sort) + if [ "${#matches[@]}" -eq 1 ]; then + printf '%s' "${matches[0]}" + return 0 + fi + if [ "${#matches[@]}" -gt 1 ]; then + local dirname="${dir##*/}" + for f in "${matches[@]}"; do + echo " $dirname/$(basename "$f")" >&2 + done + err "model dir $dirname contains multiple .gguf files — pick one (e.g. 'pos ai server start $dirname/.gguf')" + fi + return 1 +} + resolve_model() { local explicit="${1:-}" # 1. Explicit argument if [ -n "$explicit" ]; then # Absolute path if [[ "$explicit" == /* ]]; then - [ -f "$explicit" ] || err "Model not found: $explicit" - printf '%s' "$explicit" - return + [ -e "$explicit" ] || err "Model not found: $explicit" + if [ -f "$explicit" ]; then + printf '%s' "$explicit" + return + fi + # Absolute DIRECTORY — same single-.gguf expansion as below. + local abs_resolved + if abs_resolved="$(resolve_gguf_in_dir "$explicit")"; then + printf '%s' "$abs_resolved" + return + fi + err "Model not found: $explicit" fi # Relative to HF_DOWNLOAD_DIR local candidate="$HF_DOWNLOAD_DIR/$explicit" @@ -243,6 +299,17 @@ resolve_model() { printf '%s' "$candidate" return fi + # Directory under HF_DOWNLOAD_DIR (repo slug form) — the HF downloader + # writes $HF_DOWNLOAD_DIR//.gguf; a single top-level + # .gguf resolves to it, multiple err with the pick-one list, zero + # falls through to the not-found error below (F3). + if [ -d "$candidate" ]; then + local slug_resolved + if slug_resolved="$(resolve_gguf_in_dir "$candidate")"; then + printf '%s' "$slug_resolved" + return + fi + fi # Also try with the name as-is (could be a relative path) [ -f "$explicit" ] && { printf '%s' "$explicit"; return; } err "Model not found: $explicit (also searched $HF_DOWNLOAD_DIR)" @@ -301,6 +368,9 @@ Options: --metrics Enable metrics endpoint --health Enable health endpoint --slots Concurrent request slots + --no-unit Run the server directly (nohup + pidfile) instead of + installing a systemd unit — for headless/SSH boxes whose + user systemd bus is unreachable; stop/status still work Examples: pos ai server start mistral-7b-v0.1.Q4_K_M.gguf @@ -424,6 +494,8 @@ while [ $# -gt 0 ]; do --slots) [ $# -ge 2 ] || err "--slots requires a value" SLOTS="$2"; REQUESTED_FLAGS+=("--slots"); shift 2 ;; + --no-unit) + NO_UNIT=1; shift ;; -*) err "Unknown option '$1' (see --help)" ;; *) @@ -573,10 +645,47 @@ cmd_start() { exec_cmd+=" --slots $SLOTS" fi + # ── User-bus pre-flight + stale-unit warning (unit path only) ── + # ensure_user_bus aborts with remediation BEFORE the unit is written, so a + # broken user bus leaves NO orphaned unit. --no-unit skips the bus check — + # that is its whole purpose (headless/SSH boxes without a user bus). An + # existing unit is warned about but never deleted here (removal stays in + # cmd_stop); it may be stale from a previous failed start. + if [ "$NO_UNIT" -ne 1 ]; then + ensure_user_bus + if [ -f "$USER_SYSTEMD_DIR/$SERVICE" ]; then + warn "existing unit $USER_SYSTEMD_DIR/$SERVICE will be overwritten — it may be stale from a previous failed start" + fi + fi + if [ "${DRY_RUN:-0}" -eq 1 ]; then - log "(dry-run) generate systemd unit $USER_SYSTEMD_DIR/$SERVICE" - log "(dry-run) ExecStart: $exec_cmd" - log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE" + if [ "$NO_UNIT" -eq 1 ]; then + log "(dry-run) nohup $exec_cmd >$NO_UNIT_LOG 2>&1 &" + log "(dry-run) pidfile: $NO_UNIT_PIDFILE" + else + log "(dry-run) generate systemd unit $USER_SYSTEMD_DIR/$SERVICE" + log "(dry-run) ExecStart: $exec_cmd" + log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE" + fi + return 0 + fi + + # ── Direct run (--no-unit) ── + # Run the exact ExecStart command directly under nohup with a pidfile + # (under $XDG_RUNTIME_DIR, else /tmp) so stop/status keep working without + # a systemd unit. The string is the same one the unit would carry; only + # the binary and model tokens are quoted, and both are validated paths + # (command -v / existing file) resolved above, so eval is the faithful + # way to honor its systemd-style quoting. + if [ "$NO_UNIT" -eq 1 ]; then + mkdir -p "$(dirname "$NO_UNIT_PIDFILE")" + : >"$NO_UNIT_LOG" + eval "nohup $exec_cmd >'$NO_UNIT_LOG' 2>&1 &" + local direct_pid=$! + printf '%s\n' "$direct_pid" >"$NO_UNIT_PIDFILE" + log "Server starting (direct run, no systemd) — pid $direct_pid, model: $(basename "$model"), port: $PORT" + log "log: $NO_UNIT_LOG" + log "stop: kill \$(cat $NO_UNIT_PIDFILE) (or 'pos ai server stop')" return 0 fi @@ -626,8 +735,28 @@ EOF } cmd_stop() { + # Direct-run (--no-unit) pidfile first: kill the recorded pid, remove the + # pidfile. Then fall through to the normal unit path (both can exist if a + # unit was installed after a direct run). + local stopped_direct=0 + if [ -f "$NO_UNIT_PIDFILE" ]; then + stopped_direct=1 + local direct_pid + direct_pid="$(cat "$NO_UNIT_PIDFILE" 2>/dev/null || true)" + if [ "${DRY_RUN:-0}" -eq 1 ]; then + log "(dry-run) kill $direct_pid; rm -f $NO_UNIT_PIDFILE" + else + if [[ "$direct_pid" =~ ^[0-9]+$ ]]; then + kill "$direct_pid" 2>/dev/null || true + fi + rm -f "$NO_UNIT_PIDFILE" + log "llama.cpp server stopped (direct run)" + fi + fi if [ ! -f "$USER_SYSTEMD_DIR/$SERVICE" ]; then - warn "No llama.cpp server service installed ($SERVICE)" + if [ "$stopped_direct" -eq 0 ]; then + warn "No llama.cpp server service installed ($SERVICE)" + fi return 0 fi if [ "${DRY_RUN:-0}" -eq 1 ]; then @@ -647,8 +776,15 @@ cmd_status() { err "llama-server not found — install llama.cpp with the app installer: 'apps/ai/llamacpp.sh' (run 'bash apps/install.sh llamacpp', or pass '--apps'/'--full' to install.sh), see 'pos help ai server' (https://github.com/ggerganov/llama.cpp)" fi - # Service state + # Service state — systemd unit, or the direct-run (--no-unit) pidfile local svc_state="stopped" + if [ -f "$NO_UNIT_PIDFILE" ]; then + local direct_pid + direct_pid="$(cat "$NO_UNIT_PIDFILE" 2>/dev/null || true)" + if [[ "$direct_pid" =~ ^[0-9]+$ ]] && kill -0 "$direct_pid" 2>/dev/null; then + svc_state="running" + fi + fi if systemctl --user is-active "$SERVICE" &>/dev/null; then svc_state="running" fi diff --git a/bin/pos-communication-matrix-listener b/bin/pos-communication-matrix-listener index 5f55952..e1381e7 100755 --- a/bin/pos-communication-matrix-listener +++ b/bin/pos-communication-matrix-listener @@ -20,6 +20,18 @@ err() { echo "ERROR: $*" >&2; exit 1; } log() { echo "[+] $*"; } warn() { echo "[!] $*" >&2; } +# User-bus pre-flight — identical remediation to lib/common.sh's +# ensure_user_bus; this tool uses inline fallbacks instead of sourcing +# common.sh, so the helper is duplicated (guarded in case common.sh is ever +# sourced too). +declare -F ensure_user_bus >/dev/null || ensure_user_bus() { + systemctl --user show-environment &>/dev/null && return 0 + err "cannot reach the user systemd bus (common in SSH/headless sessions) — systemctl --user failed + fix: export XDG_RUNTIME_DIR=/run/user/$(id -u) (if the directory exists) + fix: sudo loginctl enable-linger $(id -un) (persist the user session so the bus survives logouts) + no unit was written — fix the bus and retry" +} + usage() { cat </dev/null || err "systemctl not found — cannot create the listener service" + # Abort BEFORE writing the unit if the user systemd bus is unreachable + # (SSH/headless) — no orphaned unit, remediation printed. + ensure_user_bus mkdir -p "$USER_SYSTEMD_DIR" local runner diff --git a/bin/pos-network-download b/bin/pos-network-download index 54f2765..98d4527 100755 --- a/bin/pos-network-download +++ b/bin/pos-network-download @@ -171,6 +171,9 @@ cmd_start() { log "(dry-run) install unit $USER_SYSTEMD_DIR/$SERVICE" log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE" else + # User-bus pre-flight BEFORE writing the unit — abort with remediation + # (no orphaned unit) when the user systemd bus is unreachable. + ensure_user_bus mkdir -p "$USER_SYSTEMD_DIR" cat > "$USER_SYSTEMD_DIR/$SERVICE" </dev/null && return 0 + err "cannot reach the user systemd bus (common in SSH/headless sessions) — systemctl --user failed + fix: export XDG_RUNTIME_DIR=/run/user/$(id -u) (if the directory exists) + fix: sudo loginctl enable-linger $(id -un) (persist the user session so the bus survives logouts) + no unit was written — fix the bus and retry" +} + # ── system.env loader ────────────────────────────────────────── # Shared "system" tool config (~/.config/linux_post_install/system.env). # Fills only variables that are not already exported — an explicitly-set diff --git a/tests/t-ai-server-bus.sh b/tests/t-ai-server-bus.sh new file mode 100644 index 0000000..6352ff9 --- /dev/null +++ b/tests/t-ai-server-bus.sh @@ -0,0 +1,225 @@ +#!/usr/bin/env bash +set -euo pipefail +# t-ai-server-bus.sh — F4 + E2E regression for the user systemd-bus pre-flight. +# +# F4 contract: +# - `pos ai server start` (unit path) calls ensure_user_bus BEFORE the unit +# is written; with no user bus (SSH/headless: XDG_RUNTIME_DIR and +# DBUS_SESSION_BUS_ADDRESS unset, systemctl --user fails) it errs with BOTH +# remediation lines and leaves NO orphaned unit file. +# - with a reachable bus the command proceeds past the pre-flight. +# - `--no-unit` is the bus-less escape hatch: it skips the pre-flight, runs +# llama-server directly under nohup with a pidfile, and stop/status can +# find it via the pidfile. +# +# E2E regression (the reported user scenario): +# - install-shaped stubs + working bus: `start Qwen-Qwen3-1.7B-GGUF` resolves +# the model dir (F3), writes a correct unit (F6: all 5 flags + --port +# 8088), reports a resolvable version (F1: 0.4.0, not "unknown"), and the +# unit passes `systemd-analyze verify`. +# - SSH-shaped no-bus env: clean pre-flight err, remediation lines, NO unit. + +run_test() { + local sandbox stubs models unitdir runtime + sandbox="$(mksandbox ai-server-bus)" + stubs="$sandbox/stubs" + models="$sandbox/models" + unitdir="$sandbox/userunits" + runtime="$sandbox/runtime" + mkdir -p "$stubs" "$models" "$unitdir" "$runtime" + + # ── User's real HF layout (F3 fixture) ── + mkdir -p "$models/Qwen-Qwen3-1.7B-GGUF" + touch "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" + + # ── Stubs ── + # llama-server: b10822-shaped (version → stderr, full-support help). + cat > "$stubs/llama-server" <<'STUB' +#!/usr/bin/env bash +case "$1" in + --version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;; + --help) + cat <<'HELP' +usage: llama-server [options] +options: + --host bind address + --port server port + --n-gpu-layers layers to offload + --ctx-size context size + --threads cpu threads +HELP + ;; + *) exit 0 ;; +esac +STUB + # systemctl: FAILS exactly like the real bus-missing case (SSH). + cat > "$stubs/systemctl" <<'STUB' +#!/usr/bin/env bash +echo "Failed to connect to user scope bus via local transport: \$DBUS_SESSION_BUS_ADDRESS and \$XDG_RUNTIME_DIR not defined" >&2 +exit 1 +STUB + # nvidia-smi: no GPU → deterministic CPU path. + printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi" + # curl: healthy JSON (health + models probes). + printf '#!/usr/bin/env bash\nprintf "%%s" '"'"'{"status":"ok"}'"'"'\n' > "$stubs/curl" + # loginctl: Linger=no (so the linger warn path is deterministic). + printf '#!/usr/bin/env bash\nprintf "Linger=no\\n"\n' > "$stubs/loginctl" + chmod +x "$stubs/llama-server" "$stubs/systemctl" "$stubs/nvidia-smi" "$stubs/curl" "$stubs/loginctl" + + local server="$ROOT/bin/pos-ai-server" + local nobus_env=(-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS + PATH="$stubs:/usr/bin:/bin" HF_DOWNLOAD_DIR="$models" + USER_SYSTEMD_DIR="$unitdir" NO_UNIT_PIDFILE="$sandbox/pos-ai-server.pid" + NO_UNIT_LOG="$sandbox/pos-ai-server.log") + local bus_env=(-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS + PATH="$stubs:/usr/bin:/bin" HF_DOWNLOAD_DIR="$models" + USER_SYSTEMD_DIR="$unitdir" XDG_RUNTIME_DIR="$runtime" + DBUS_SESSION_BUS_ADDRESS="unix:path=$runtime/bus" + NO_UNIT_PIDFILE="$sandbox/pos-ai-server.pid" NO_UNIT_LOG="$sandbox/pos-ai-server.log") + + # ═══ F4 pre-flight: bus missing → err + remediation + NO unit ═══ + test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF + check_contains "F4 bus-missing start names the bus problem" "user systemd bus" "$TR_OUT" + check_contains "F4 remediation: export XDG_RUNTIME_DIR" "export XDG_RUNTIME_DIR=/run/user/" "$TR_OUT" + check_contains "F4 remediation: sudo loginctl enable-linger" "sudo loginctl enable-linger" "$TR_OUT" + check_contains "F4 states no unit was written" "no unit was written" "$TR_OUT" + [ "$TR_RC" -ne 0 ] && printf ' PASS F4 bus-missing start exits nonzero\n' || printf ' FAIL F4 bus-missing start exited 0\n' + check_file_absent "F4 no orphaned unit written" "$unitdir/pos-ai-server.service" + + # ═══ F4 pre-flight: bus present → proceeds past pre-flight (DRY_RUN) ═══ + # Replace the failing systemctl with a succeeding one. + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/systemctl" + test_run_env "${bus_env[@]}" DRY_RUN=1 -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF + check_rc "F4 bus-present DRY_RUN start exits 0" 0 "$TR_RC" + check_contains "F4 bus-present proceeds past pre-flight (ExecStart)" "ExecStart:" "$TR_OUT" + + # ═══ F4 --no-unit escape hatch (bus-less direct run) ═══ + # A real --no-unit start must skip the bus check and launch the stub under + # nohup with a pidfile; status finds it; stop kills it. The stub sleeps via + # `exec sleep` so the recorded pid IS the sleeper (no orphan children). + cat > "$stubs/llama-server" <<'STUB' +#!/usr/bin/env bash +case "$1" in + --version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;; + --help) + cat <<'HELP' +usage: llama-server [options] +options: + --host bind address + --port server port + --n-gpu-layers layers to offload + --ctx-size context size + --threads cpu threads +HELP + ;; + *) exec sleep 300 ;; +esac +STUB + chmod +x "$stubs/llama-server" + + # failing systemctl stub again: --no-unit must NOT care. + cat > "$stubs/systemctl" <<'STUB' +#!/usr/bin/env bash +echo "no bus" >&2 +exit 1 +STUB + chmod +x "$stubs/systemctl" + + test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF --no-unit + check_rc "F4 --no-unit bus-less start exits 0" 0 "$TR_RC" + check_file_exists "F4 --no-unit writes pidfile" "$sandbox/pos-ai-server.pid" + check_contains "F4 --no-unit prints log path hint" "log:" "$TR_OUT" + local direct_pid + direct_pid="$(cat "$sandbox/pos-ai-server.pid" 2>/dev/null || true)" + if [[ "$direct_pid" =~ ^[0-9]+$ ]] && kill -0 "$direct_pid" 2>/dev/null; then + printf ' PASS F4 --no-unit launched a live server process (pid %s)\n' "$direct_pid" + else + printf ' FAIL F4 --no-unit pid %s not alive\n' "${direct_pid:-EMPTY}" + fi + + # status via the pidfile reports running (deps + curl stubs present). + test_run_env "${nobus_env[@]}" -- timeout 30 "$server" status + check_rc "F4 status exits 0 after --no-unit start" 0 "$TR_RC" + check_contains "F4 status finds direct-run server running" "service: running" "$TR_OUT" + + # stop kills the pid and removes the pidfile. + test_run_env "${nobus_env[@]}" -- timeout 30 "$server" stop + check_rc "F4 --no-unit stop exits 0" 0 "$TR_RC" + check_file_absent "F4 stop removes pidfile" "$sandbox/pos-ai-server.pid" + if [[ "$direct_pid" =~ ^[0-9]+$ ]] && ! kill -0 "$direct_pid" 2>/dev/null; then + printf ' PASS F4 stop killed the direct-run process\n' + else + printf ' FAIL F4 direct-run process still alive after stop\n' + fi + + # ═══ E2E positive: user scenario with a working bus ═══ + # Restore full-support stub + succeeding systemctl. + cat > "$stubs/llama-server" <<'STUB' +#!/usr/bin/env bash +case "$1" in + --version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;; + --help) + cat <<'HELP' +usage: llama-server [options] +options: + --host bind address + --port server port + --n-gpu-layers layers to offload + --ctx-size context size + --threads cpu threads +HELP + ;; + *) exit 0 ;; +esac +STUB + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/llama-server" "$stubs/systemctl" + + test_run_env "${bus_env[@]}" -- timeout 60 "$server" start Qwen-Qwen3-1.7B-GGUF + check_rc "E2E user scenario start exits 0" 0 "$TR_RC" + check_file_exists "E2E unit written" "$unitdir/pos-ai-server.service" + + local unit="$unitdir/pos-ai-server.service" + local unit_exec + unit_exec="$(grep '^ExecStart=' "$unit")" + check_contains "E2E unit: model expanded from dir" "Qwen3-1.7B-Q8_0.gguf" "$unit_exec" + check_contains "E2E unit: --port 8088 pinned" "--port 8088" "$unit_exec" + check_contains "E2E unit: --host present" "--host 127.0.0.1" "$unit_exec" + check_contains "E2E unit: --n-gpu-layers present" "--n-gpu-layers 0" "$unit_exec" + check_contains "E2E unit: --ctx-size present" "--ctx-size 4096" "$unit_exec" + check_contains "E2E unit: --threads present" "--threads" "$unit_exec" + + # Version resolvable through the real CLI on the same run's stub. + test_run_env "${bus_env[@]}" -- timeout 30 "$server" status + check_contains "E2E version is resolvable (0.4.0)" "version: 0.4.0" "$TR_OUT" + check_not_contains "E2E version is NOT unknown" "version: unknown" "$TR_OUT" + + # Unit must pass systemd-analyze verify. + if command -v systemd-analyze >/dev/null 2>&1; then + test_run systemd-analyze verify "$unit" + check_rc "E2E systemd-analyze verify accepts unit" 0 "$TR_RC" + else + skip_case "E2E systemd-analyze verify" "systemd-analyze not available" + fi + + # stop cleans up (unit removed). + test_run_env "${bus_env[@]}" -- timeout 30 "$server" stop + check_rc "E2E stop exits 0" 0 "$TR_RC" + check_file_absent "E2E stop removes unit" "$unitdir/pos-ai-server.service" + + # ═══ E2E SSH-shaped: no bus → clean pre-flight err, remediation, NO unit ═══ + cat > "$stubs/systemctl" <<'STUB' +#!/usr/bin/env bash +echo "Failed to connect to user scope bus via local transport: \$DBUS_SESSION_BUS_ADDRESS and \$XDG_RUNTIME_DIR not defined" >&2 +exit 1 +STUB + chmod +x "$stubs/systemctl" + test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF + check_rc "E2E SSH-shaped start exits nonzero" 1 "$TR_RC" + check_contains "E2E SSH-shaped: bus problem named" "user systemd bus" "$TR_OUT" + check_contains "E2E SSH-shaped: export XDG_RUNTIME_DIR remediation" "export XDG_RUNTIME_DIR=/run/user/" "$TR_OUT" + check_contains "E2E SSH-shaped: loginctl enable-linger remediation" "sudo loginctl enable-linger" "$TR_OUT" + check_contains "E2E SSH-shaped: no unit was written" "no unit was written" "$TR_OUT" + check_file_absent "E2E SSH-shaped: no orphaned unit" "$unitdir/pos-ai-server.service" +} \ No newline at end of file diff --git a/tests/t-ai-server-flags.sh b/tests/t-ai-server-flags.sh index 3f9b6e5..467a017 100644 --- a/tests/t-ai-server-flags.sh +++ b/tests/t-ai-server-flags.sh @@ -40,7 +40,11 @@ STUB #!/usr/bin/env bash exit 1 STUB - chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" + # systemctl: succeed — F4's ensure_user_bus pre-flight must pass so the + # dry-run reaches the ExecStart output this test asserts on (bus-missing + # behavior is covered separately in t-ai-server-bus.sh). + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" "$stubs/systemctl" local server="$ROOT/bin/pos-ai-server" local base_env=(PATH="$stubs:/usr/bin:/bin" DRY_RUN=1 diff --git a/tests/t-ai-server-model.sh b/tests/t-ai-server-model.sh new file mode 100644 index 0000000..eac8884 --- /dev/null +++ b/tests/t-ai-server-model.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +set -euo pipefail +# t-ai-server-model.sh — F3: `resolve_model` directory expansion regression. +# The HF downloader writes weights as `$HF_DOWNLOAD_DIR//.gguf`, +# but the old resolve_model accepted only FILE paths, so the user's +# `pos ai server start Qwen-Qwen3-1.7B-GGUF` failed with "Model not found". +# F3 adds: dir with exactly one .gguf → resolve; multiple → err listing each +# as `/` + "pick one"; zero → fall through to not-found. It also +# accepts the `/.gguf` slug/file form. Flat-file + absolute paths +# must keep working (backward compatibility). +# +# Unit level: call the REAL shipped resolve_model / resolve_gguf_in_dir with a +# sandbox HF_DOWNLOAD_DIR. Integration level: `start` DRY_RUN resolves the +# user's real dir name. + +# extract_fn — print one brace-delimited function body. +extract_fn() { + local file="$1" fn="$2" + awk -v fn="$fn" ' + BEGIN { found=0; depth=0 } + { + if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 } + if (found) { + n_open = gsub(/\{/, "{") + n_close = gsub(/\}/, "}") + depth = depth + n_open - n_close + print + if (depth <= 0) exit + } + } + ' "$file" +} + +run_test() { + source "$ROOT/lib/common.sh" # err/log for resolve_model + + local sandbox models aid + sandbox="$(mksandbox ai-server-model)" + models="$sandbox/models" + mkdir -p "$models" + aid="$ROOT/bin/pos-ai-server" + + # Extract the REAL model-resolution functions. + local fn_file="$sandbox/fns.sh" com="$ROOT/lib/common.sh" + { + extract_fn "$aid" resolve_gguf_in_dir + extract_fn "$aid" resolve_model + } > "$fn_file" + + # ── Fixtures (mirror the user's real layout) ── + # single-gguf dir (the reported case): Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf + mkdir -p "$models/Qwen-Qwen3-1.7B-GGUF" + touch "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" + # multi-gguf dir + mkdir -p "$models/multi" + touch "$models/multi/q1.gguf" "$models/multi/q2.gguf" + # empty dir + mkdir -p "$models/empty" + # flat-file legacy form + touch "$models/flat.gguf" + # absolute file + touch "$models/abs.gguf" + + # ── F3 unit cases (direct function calls) ── + # 1. single-gguf dir resolves to the inner file + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'Qwen-Qwen3-1.7B-GGUF'" + check_rc "F3 single-gguf dir resolves (rc 0)" 0 "$TR_RC" + check_eq "F3 resolves to inner gguf" "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" "$TR_OUT" + + # 2. multi-gguf dir errors listing each as / + "pick one" + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'multi'" + check_rc "F3 multi-gguf dir errors (rc 1)" 1 "$TR_RC" + check_contains "F3 multi lists q1" "multi/q1.gguf" "$TR_OUT" + check_contains "F3 multi lists q2" "multi/q2.gguf" "$TR_OUT" + check_contains "F3 multi says pick one" "pick one" "$TR_OUT" + + # 3. zero-gguf dir → falls through to today's not-found + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'empty'" + check_rc "F3 empty dir errors (rc 1)" 1 "$TR_RC" + check_contains "F3 empty dir -> Model not found" "Model not found" "$TR_OUT" + + # 4. slug/file form /.gguf resolves + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf'" + check_rc "F3 slug/file form resolves (rc 0)" 0 "$TR_RC" + check_eq "F3 slug/file resolves to inner gguf" "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" "$TR_OUT" + + # 5. flat-file legacy form still works + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'flat.gguf'" + check_rc "F3 flat file resolves (rc 0)" 0 "$TR_RC" + check_eq "F3 flat file stays flat (backward compat)" "$models/flat.gguf" "$TR_OUT" + + # 6. absolute file path still works + test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model '$models/abs.gguf'" + check_rc "F3 absolute file resolves (rc 0)" 0 "$TR_RC" + check_eq "F3 absolute file unchanged" "$models/abs.gguf" "$TR_OUT" + + # ── F3 integration: `start ` DRY_RUN resolves through the full CLI ── + local stubs="$sandbox/stubs" + mkdir -p "$stubs" + cat > "$stubs/llama-server" <<'STUB' +#!/usr/bin/env bash +case "$1" in + --version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;; + --help) + cat <<'HELP' +usage: llama-server [options] +options: + --host bind address + --port server port + --n-gpu-layers layers to offload + --ctx-size context size + --threads cpu threads +HELP + ;; +esac +STUB + printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi" + chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" + + local server="$ROOT/bin/pos-ai-server" + test_run_env PATH="$stubs:/usr/bin:/bin" DRY_RUN=1 \ + HF_DOWNLOAD_DIR="$models" USER_SYSTEMD_DIR="$sandbox/userunits" -- \ + "$server" start Qwen-Qwen3-1.7B-GGUF --no-unit + check_rc "F3 CLI start with dir arg exits 0" 0 "$TR_RC" + check_contains "F3 CLI resolves dir to inner file" \ + "-m \"$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf\"" "$TR_OUT" + + # multi-gguf via CLI → clean err naming pick-one, rc 1 + test_run_env PATH="$stubs:/usr/bin:/bin" DRY_RUN=1 \ + HF_DOWNLOAD_DIR="$models" USER_SYSTEMD_DIR="$sandbox/userunits" -- \ + "$server" start multi --no-unit + check_rc "F3 CLI multi-gguf start errors (rc 1)" 1 "$TR_RC" + check_contains "F3 CLI multi-gguf says pick one" "pick one" "$TR_OUT" +} diff --git a/tests/t-ai-server-validate.sh b/tests/t-ai-server-validate.sh new file mode 100644 index 0000000..188e096 --- /dev/null +++ b/tests/t-ai-server-validate.sh @@ -0,0 +1,241 @@ +#!/usr/bin/env bash +set -euo pipefail +# t-ai-server-validate.sh — F1/F2/F5/F6 regression for `pos ai server`: +# F1 detect_llama_version reads llama.cpp's version string from STDERR +# (real builds print `version: 0.4.0-dev (build 10822, …)` to stderr), +# and falls back to "unknown" (no errexit) when unreadable/missing. +# F2 flag validation is pipe-less (no printf|grep -q SIGPIPE/rc=141 race): +# validate_default_flags / validate_requested_flags are deterministic — +# all 5 defaults kept every run on a real-shaped ~59 KB help, and a +# requested flag is always accepted/rejected the same way. +# F5 find_llamacpp no longer matches an unrelated bare `server` binary. +# F6 --port $PORT (and the 4 other defaults) always appear in ExecStart. +# +# Strategy: F1/F5/F2-internals call the REAL shipped functions directly +# (brace-matching extraction, so production logic is never re-typed); F6 +# exercises the full CLI with the real-shaped help fixture. + +# extract_fn — print one brace-delimited function body. +# Used to unit-test a self-contained function that lives in a thick script +# whose dispatch we must not trigger. +extract_fn() { + local file="$1" fn="$2" + awk -v fn="$fn" ' + BEGIN { found=0; depth=0 } + { + if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 } + if (found) { + n_open = gsub(/\{/, "{") + n_close = gsub(/\}/, "}") + depth = depth + n_open - n_close + print + if (depth <= 0) exit + } + } + ' "$file" +} + +# ── F1/F2 help fixture: 732-line ~59 KB body carrying the 5 defaults at the +# REAL llama.cpp line offsets (7/25/140/503/506) — the shape that used to +# trip the printf|grep -q SIGPIPE race under pipefail. +build_real_help() { + local total=732 i + for ((i = 1; i <= total; i++)); do + case "$i" in + 7) printf '%s\n' ' -t, --threads N' ;; + 25) printf '%s\n' ' -c, --ctx-size N' ;; + 140) printf '%s\n' ' -ngl, --gpu-layers, --n-gpu-layers N' ;; + 503) printf '%s\n' '--host HOST' ;; + 506) printf '%s\n' '--port PORT' ;; + *) + if [ "$i" -le 2 ]; then + printf 'usage: llama-server [options]\n' + else + printf ' --general-option-%04d example option number %d with a longish description\n' "$i" "$i" + fi + ;; + esac + done +} + +new_stub() { # new_stub — full-support llama-server stub (stderr version) + local dir="$1" help_fixture + help_fixture="$(build_real_help)" + cat > "$dir/llama-server" <&2 ;; + --help) + cat <<'EOF_HELP' +$help_fixture +EOF_HELP + ;; + *) exit 0 ;; +esac +STUB + chmod +x "$dir/llama-server" +} + +run_test() { + source "$ROOT/lib/common.sh" # err/warn/log for the extracted functions + + local sandbox stubs + sandbox="$(mksandbox ai-server-validate)" + stubs="$sandbox/stubs" + mkdir -p "$stubs" + + local aid="$ROOT/bin/pos-ai-server" + local fn_file="$sandbox/fns.sh" + + # ═══ F1: detect_llama_version (STDERR capture + "unknown" guard) ═══ + extract_fn "$aid" detect_llama_version > "$fn_file" + source "$fn_file" + + cat > "$stubs/f1-stderr" <<'STUB' +#!/usr/bin/env bash +printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 +STUB + chmod +x "$stubs/f1-stderr" + test_run detect_llama_version "$stubs/f1-stderr" + check_rc "F1 stderr semver -> rc 0" 0 "$TR_RC" + check_eq "F1 stderr semver resolved (not unknown)" "0.4.0" "$TR_OUT" + + cat > "$stubs/f1-build" <<'STUB' +#!/usr/bin/env bash +printf 'version: build 10822\n' >&2 +STUB + chmod +x "$stubs/f1-build" + test_run detect_llama_version "$stubs/f1-build" + check_rc "F1 build-only -> rc 0" 0 "$TR_RC" + check_eq "F1 build 10822 -> 10822" "10822" "$TR_OUT" + + cat > "$stubs/f1-b" <<'STUB' +#!/usr/bin/env bash +printf 'version: b10822\n' >&2 +STUB + chmod +x "$stubs/f1-b" + test_run detect_llama_version "$stubs/f1-b" + check_eq "F1 b10822 -> b10822" "b10822" "$TR_OUT" + + # missing binary → "unknown", rc 0, NO errexit. + test_run detect_llama_version "$sandbox/does-not-exist-llama-server" + check_rc "F1 missing binary -> rc 0 (no errexit)" 0 "$TR_RC" + check_eq "F1 missing binary -> unknown" "unknown" "$TR_OUT" + + # binary exists but --version prints nothing parseable → unknown. + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/f1-noop" + chmod +x "$stubs/f1-noop" + test_run detect_llama_version "$stubs/f1-noop" + check_eq "F1 unreadable version -> unknown" "unknown" "$TR_OUT" + + # ═══ F5: find_llamacpp candidate narrowing ═══ + extract_fn "$aid" find_llamacpp > "$fn_file" + source "$fn_file" + + local onlyserver="$sandbox/onlyserver" + mkdir -p "$onlyserver" + ln -s /usr/bin/false "$onlyserver/server" + ( + PATH="$onlyserver:/usr/bin:/bin" + test_run find_llamacpp + printf '%s' "$TR_RC" > "$sandbox/f5.rc" + printf '%s' "$TR_OUT" > "$sandbox/f5.out" + ) + check_rc "F5 bare 'server' alone NOT found (rc != 0)" 1 "$(cat "$sandbox/f5.rc")" + check_eq "F5 'server' candidate yields nothing" "" "$(cat "$sandbox/f5.out")" + + # llama-server IS a legitimate candidate; found first (rc 0 name). + local havebin="$sandbox/havebin" + mkdir -p "$havebin" + new_stub "$havebin" + ( + PATH="$havebin:/usr/bin:/bin" + test_run find_llamacpp + printf '%s' "$TR_RC" > "$sandbox/f5b.rc" + printf '%s' "$TR_OUT" > "$sandbox/f5b.out" + ) + check_rc "F5 llama-server present -> found (rc 0)" 0 "$(cat "$sandbox/f5b.rc")" + check_eq "F5 returns llama-server (not server)" "llama-server" "$(cat "$sandbox/f5b.out")" + + # ═══ F2: deterministic flag validation (pipe-less, no rc=141) ═══ + # Extract both validation functions; they use err/warn from common.sh. + { + extract_fn "$aid" validate_requested_flags + extract_fn "$aid" validate_default_flags + } > "$fn_file" + local com="$ROOT/lib/common.sh" + + # Default-flag determinism: 25 runs over the real-shaped help; every run + # must keep all 5 defaults (the pre-fix SIGPIPE race dropped a random + # subset per run). Run in a subshell sourcing the exact function bodies + + # common.sh, capturing the globals it sets. + local run stable=1 gvals run_rc + for run in $(seq 1 25); do + gvals="$(bash -c " + source '$com' + source '$fn_file' + validate_default_flags '$havebin/llama-server' 0.4.0 '' + printf '%s|%s|%s|%s|%s' \"\${DEFAULT_PORT_OK}\" \"\${DEFAULT_HOST_OK}\" \\ + \"\${DEFAULT_GPU_OK}\" \"\${DEFAULT_CTX_OK}\" \"\${DEFAULT_THREADS_OK}\" + " 2>/dev/null)" + run_rc=$? + if [ "$run_rc" -ne 0 ]; then + stable=0 + printf ' FAIL F2 default-flag validation run %s: whole invocation rc=%s (possible rc=141 SIGPIPE component)\n' "$run" "$run_rc" + break + fi + if [ "$gvals" != "1|1|1|1|1" ]; then + stable=0 + printf ' FAIL F2 run %s dropped a default (globals=%s)\n' "$run" "$gvals" + break + fi + done + [ "$stable" -eq 1 ] && printf ' PASS F2 validate_default_flags keeps all 5 defaults across 25 pipefail runs (whole-invocation rc=0 every run)\n' + + # Requested-flag determinism (20+ runs each): supported always rc 0, + # unsupported always hard-errors (never a flaky pass). + local ok=0 bad=0 + for run in $(seq 1 20); do + if bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --port" 2>/dev/null; then + ok=$((ok + 1)) + fi + if bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --tensor-split 1" 2>/dev/null; then + bad=$((bad + 1)) + fi + done + check_eq "F2 requested supported --port accepted every run" 20 "$ok" + check_eq "F2 no flaky pass on unsupported requested flag" 0 "$bad" + + test_run bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --tensor-split 1" + check_rc "F2 unsupported requested flag exits 1" 1 "$TR_RC" + check_contains "F2 unsupported requested flag names it" "--tensor-split" "$TR_OUT" + check_contains "F2 unsupported requested flag is version-aware" "0.4.0" "$TR_OUT" + + # ═══ F6: full CLI — port pinning + version not "unknown" ═══ + local models="$sandbox/models" + mkdir -p "$models" + touch "$models/my-model.gguf" + printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi" + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/nvidia-smi" "$stubs/systemctl" + + local server="$ROOT/bin/pos-ai-server" + local base_env=(PATH="$havebin:$stubs:/usr/bin:/bin" DRY_RUN=1 + USER_SYSTEMD_DIR="$sandbox/userunits") + test_run_env "${base_env[@]}" -- "$server" start "$models/my-model.gguf" + check_rc "F6 DRY_RUN start exits 0" 0 "$TR_RC" + local exec_line + exec_line="$(printf '%s\n' "$TR_OUT" | grep 'dry-run) ExecStart:' | sed 's/.*ExecStart: //')" + check_contains "F6 --port 8088 pinned in ExecStart" "--port 8088" "$exec_line" + check_contains "F6 --host present" "--host 127.0.0.1" "$exec_line" + check_contains "F6 --n-gpu-layers present" "--n-gpu-layers 0" "$exec_line" + check_contains "F6 --ctx-size present" "--ctx-size 4096" "$exec_line" + check_contains "F6 --threads present" "--threads" "$exec_line" + + # Version via the same b10822-shaped stub: status shows 0.4.0, never + # "unknown" (F1 proven through the real CLI on a real-shaped build). + test_run_env "${base_env[@]}" -- "$server" status + check_rc "F1 CLI status exits 0" 0 "$TR_RC" + check_contains "F1 CLI status resolves version 0.4.0" "version: 0.4.0" "$TR_OUT" + check_not_contains "F1 CLI status version is NOT unknown" "version: unknown" "$TR_OUT" +} diff --git a/tests/t-config-precedence.sh b/tests/t-config-precedence.sh index e603ee3..36f0562 100644 --- a/tests/t-config-precedence.sh +++ b/tests/t-config-precedence.sh @@ -122,7 +122,11 @@ HELP esac STUB printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi" - chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" + # systemctl: succeed — F4's ensure_user_bus pre-flight must pass so Part B's + # dry-run reaches the ExecStart output it asserts on (bus-missing behavior + # lives in t-ai-server-bus.sh, not here). + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" "$stubs/systemctl" local models="$sandbox/models" mkdir -p "$models" diff --git a/tests/t-llamacpp-install.sh b/tests/t-llamacpp-install.sh new file mode 100644 index 0000000..61f9f2a --- /dev/null +++ b/tests/t-llamacpp-install.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +set -euo pipefail +# t-llamacpp-install.sh — F7: `apps/ai/llamacpp.sh` post-install sanity. +# After the installer drops llama-server + symlinks it into the bin dir, the +# sanity check must verify the binary actually runs (version/help readable via +# the same stdout/stderr the tool chain reads) and the symlink is not dangling. +# healthy → passes, logs "llama.cpp sanity OK" +# dangling symlink → err (rc 1, "dangling symlink") +# non-executable → err (rc 1, not on PATH) +# broken binary → err (rc 1, --version did not run → missing shared lib) +# Uses the LLAMACPP_BIN_DIR seam; calls the REAL shipped function. +# +# Note: the real installer cannot run in the sandbox (network + sudo + root +# paths), so the sanity function is extracted verbatim and exercised directly. + +# extract_fn — print one brace-delimited function body. +extract_fn() { + local file="$1" fn="$2" + awk -v fn="$fn" ' + BEGIN { found=0; depth=0 } + { + if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 } + if (found) { + n_open = gsub(/\{/, "{") + n_close = gsub(/\}/, "}") + depth = depth + n_open - n_close + print + if (depth <= 0) exit + } + } + ' "$file" +} + +run_test() { + local app="$ROOT/apps/ai/llamacpp.sh" + local com="$ROOT/lib/common.sh" + + # Extract the REAL llamacpp_sanity from the shipped installer. + local fn_file="$TEST_TMP/llamacpp-fns.sh" + extract_fn "$app" llamacpp_sanity > "$fn_file" + + local sandbox bindir + sandbox="$(mksandbox llamacpp-install)" + bindir="$sandbox/bin" + mkdir -p "$bindir" + + # 1. healthy fixture → rc 0 + "llama.cpp sanity OK" + cat > "$bindir/llama-server" <<'STUB' +#!/usr/bin/env bash +# healthy: version + help both readable; llama.cpp prints version to stderr +printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 +exit 0 +STUB + chmod +x "$bindir/llama-server" + test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \ + bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1 + check_rc "F7 healthy fixture sanity rc 0" 0 "$TR_RC" + check_contains "F7 healthy fixture sanity OK log" "llama.cpp sanity OK" "$TR_OUT" + + # 2. dangling symlink → err, rc 1, names the broken link + rm -f "$bindir/llama-server" + ln -s "$bindir/no-such-target" "$bindir/llama-server" + test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \ + bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1 + check_rc "F7 dangling symlink errs (rc 1)" 1 "$TR_RC" + check_contains "F7 dangling symlink message" "dangling symlink" "$TR_OUT" + + # 3. non-executable fixture → err, rc 1 (bash `command -v` finds the file + # by PATH existence; the binary fails when actually executed, so the + # sanity errs at the --version step — the contract is a non-zero err). + rm -f "$bindir/llama-server" + printf '#!/usr/bin/env bash\nexit 0\n' > "$bindir/llama-server" + chmod 644 "$bindir/llama-server" # NOT executable + test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \ + bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1 + check_rc "F7 non-executable errs (rc 1)" 1 "$TR_RC" + check_contains "F7 non-executable message is a sanity err" "llama.cpp install sanity failed" "$TR_OUT" + + # 4. broken binary (--version fails like a missing shared lib) → err, rc 1 + rm -f "$bindir/llama-server" + cat > "$bindir/llama-server" <<'STUB' +#!/usr/bin/env bash +# simulates a truncated/missing-shared-lib binary: version unreadable +echo "error while loading shared libraries: libllama.so: cannot open" >&2 +exit 1 +STUB + chmod +x "$bindir/llama-server" + test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \ + bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1 + check_rc "F7 broken binary errs (rc 1)" 1 "$TR_RC" + check_contains "F7 broken binary message" "did not run" "$TR_OUT" + + # 5. static guard: the shipped installer actually CALLS the sanity after + # the symlink loop (the wiring that makes F7 real). + if grep -q 'llamacpp_sanity' "$app"; then + printf ' PASS F7 installer wires llamacpp_sanity into install\n' + else + printf ' FAIL F7 installer does not call llamacpp_sanity\n' + fi +} \ No newline at end of file diff --git a/tests/t-unsupported-flags.sh b/tests/t-unsupported-flags.sh index 8d83dfc..09ffd4e 100644 --- a/tests/t-unsupported-flags.sh +++ b/tests/t-unsupported-flags.sh @@ -43,6 +43,11 @@ HELP esac STUB chmod +x "$stubs/llama-server" + # systemctl: succeed — F4's ensure_user_bus pre-flight must pass so these + # flag-validation checks reach their intended outcome (unsupported-flag / + # default-flag behavior). Bus-missing behavior lives in t-ai-server-bus.sh. + printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl" + chmod +x "$stubs/systemctl" # 1. CLI-explicit unsupported flag → rc 1, names flag + model version test_run_env "${base_env[@]}" -- "$server" start "$models/my-model.gguf" --tensor-split 1:2:3