fix: llama-server start breakage — version detection, flag-validation race, model dir resolution, user-bus pre-flight
gates / consistency-and-conventions (push) Successful in 26s
gates / consistency-and-conventions (push) Successful in 26s
User report after the llamacpp app install: 'installed llama.cpp unknown', valid flags rejected (randomly per run), 'Model not found' for the HF downloader's own layout, and a systemd user-bus failure over SSH. Detective (real b10822 binary, FACT) found four independent causes: - version: llama-server --version prints to STDERR; detect_llama_version's 2>/dev/null swallowed it -> always 'unknown'. Now captures 2>&1 + accepts semver/build tokens (incl. build 1.2.3 edge) - validation: printf|grep -q under pipefail -> SIGPIPE rc=141 race randomly rejected flags present in the 59 KB --help. Now pipe-less grep (no race); 20x determinism regression test - model resolution: resolve_model accepted files only, but the HF downloader creates <models>/<repo>/file.gguf dirs. Now expands a dir with exactly one *.gguf (never silently picks; multi-gguf lists + errs) - port: llama.cpp default 8080 vs tool/adapter 8088; validation reliability means --port is now always pinned in the unit - user bus: headless/SSH sessions lack XDG_RUNTIME_DIR -> ensure_user_bus in lib/common.sh pre-flights all three systemctl --user tools with remediation text; pos ai server --no-unit direct-run escape hatch (pidfile) for boxes with no bus - find_llamacpp narrowed to llama-server/llama-server-cuda (bare 'server' fallback hazard); installer post-install sanity (version+help execute, symlink targets resolve) Architect decisions DQ1-DQ6 recorded. Tester: 4 new regression files (version-from-stderr, 25x flag-validation determinism, model dir expansion, bus pre-flight + E2E) + 3 fixture updates; suite 16 files / 269 checks. Verified: make gen idempotent; make check OK; make lint 0 FAIL, 0 WARN; make test 269/269 (~49s); bash -n clean; git diff --check clean.
This commit is contained in:
Your Name
@@ -42,6 +42,8 @@ summary (newest last).
|
||||
|
||||
## Done
|
||||
|
||||
- **2026-09-06** — AI server breakage post-llamacpp install — four root causes found and fixed (Detective→Architect→Builder→Tester chain). (1) `llama-server --version` prints to STDERR — `detect_llama_version`'s `2>/dev/null` swallowed it → "installed llama.cpp unknown"; (2) `printf|grep -q` under `pipefail` → SIGPIPE rc=141 race randomly rejecting valid flags from the 59 KB `--help`; (3) `resolve_model` expected flat files but the HF downloader creates `<models>/<repo>/file.gguf` dirs → `Model not found`; (4) llama.cpp default port 8080 vs tool's 8088. Architect DQ1-DQ6: help-gated validation stays; dir-expansion never silently picks; `ensure_user_bus` in `lib/common.sh` pre-flights all three tools; `--no-unit` direct-run escape hatch for SSH/headless; candidates narrowed to `llama-server`/`llama-server-cuda`; port pinned 8088; installer post-install sanity. Builder F1-F7 (stderr version capture, pipefail-safe flag validation, model dir expansion, port hardcoding, `--no-unit`, user-bus pre-flight, installer sanity + F1 regex edge: `build 1.2.3`→`1` misparse). Tester: 4 new regression files (version-from-stderr, 20× flag-validation determinism, model dir expansion, bus pre-flight + E2E) + 3 fixture updates; suite now 16 files / 269 checks. Verified: `make gen` idempotent, `make check` OK, `make lint` 0 FAIL / 0 WARN, `make test` 269/269 (~49 s), `bash -n` clean, `git diff --check` clean. Post-fix: user's machine needed only `export XDG_RUNTIME_DIR=/run/user/1000` (linger already on) → Option A systemd-managed server works, or `--no-unit` for direct run.
|
||||
|
||||
- **2026-09-06** — Stabilization pass: 17-point code-level audit executed via Explorer(3) → Architect(decisions D-A..D-F) → Builder(ai/security/tooling/config/netprobe/f1-f3) → Tester(regression suite) → Reviewer(2 rounds). Security: Telegram sender-owner AND-gate + `TELEGRAM_OWNER_ID` registry/docs; Matrix `MATRIX_ROOM_ID` required; gpg `--passphrase-fd 3` (no argv secret); `/dev/tcp` positional-arg form (checkport/smb-client/share-lib/NET_PROBE incl. escaping `\$1`/`\$2`); eval `--no-command-execution` now carried by both chat bridges (D-B), deny-by-default `[y/N]`, tty-gated `--trust`; D-A soft-fail model ratified by Architect amendment (fail-closed either way; listeners are `Restart=always` so strict mode would crash-loop). AI: ALL ExecStart flags validated against installed llama.cpp (requested→error, default→omit+warn, `CONFIG_REQUESTED_FLAGS`), single-line ExecStart confirmed via `systemd-analyze verify`; hf single-file failure rc/exit-0 + meta-write bug fixed; `LLAMACPP_HOST` coherent; `# POS_SUBCMDS` + metadata gaps closed. Tooling: lint-conventions rewritten Bash-native (~24-30× faster, rules byte-identical, `:num` restored on 2 WARNs, planted-violation negative verified); `pos system uninstall` covers all 12 libs + scale-tail + flags dir + systemd USER units (`|| true`) + de-hardcoded plugin markers; safe anchored `.bash_completion`/`.bashrc` removals replace `sed -i '/pos/d'`. Config: canonical `load_env_file` in `lib/config-ui.sh` (CRLF strip, env-wins, XDG, `LOADED_ENV_KEYS`); 9 tools migrated; entertainment-lib collapsed to wrappers; docker-compose deliberately NOT migrated (source-semantics, documented). Tests: first committed regression suite — `tests/run-tests.sh` zero-dep runner + `make test`; 12 files / 179 checks / 0 skip / ~52s; hard skip contract (never lie); negative lint/gen-drift gates; systemd-analyze verify included. Verification: `make gen` idempotent, `make check` OK, `make lint` `0 FAIL, 0 WARN`, `make test` green, `bash -n` clean, `git diff --check` clean; `systemd-analyze verify` PASS on generated unit; CLI smokes (`pos --help`, `pos ai --help`, `pos ai hf --help`, `pos ai server --help`, `pos tree`) OK.
|
||||
|
||||
- **2026-09-06** — llamacpp optional-app installer + `ai` app category + `pos ai server` install-hint wiring: new `apps/ai/llamacpp.sh` (idempotent `install_llamacpp()`/`uninstall_llamacpp()` — GitHub release archive: scans `/releases?per_page=10` for the first `-bin-ubuntu-{x64,arm64}.tar.gz` asset since `releases/latest` (v0.4.0 milestone) ships no binaries; installs to `/usr/local/lib/llama.cpp-<tag>` with `--strip-components=1`, symlinks every `llama*` binary into `/usr/local/bin`; uninstall removes the lib dir and only the symlinks whose target points into it); `apps/install.sh` CAT_NAMES += `[ai]="AI / ML"`; `templates/app.sh` categories comment += ai; DOC/APPS.md app-table row count updated (15→16 at the time; 18 after the 2026-09-06 stabilization pass) + categories line + llama.cpp catalog row; `bin/pos-ai-server` help "Requires:" + both `err "llama-server not found…"` lines (start/status) now name the app installer + invocation (`bash apps/install.sh llamacpp`, `--apps`, `--full`) and keep the GitHub URL (scrcpy phrasing pattern); DOC/POS.md ai row notes `bash apps/install.sh llamacpp`. Verified: live API probe confirmed real asset naming — `llama-<tag>-bin-ubuntu-x64.tar.gz`/`-arm64.tar.gz` on nightly bNNNNN releases, top-level dir present, `.tar.gz` not `.zip` (so `tar` replaces the brief's `unzip` step — no unnecessary apt install); `bash -n` touched scripts; `bash apps/install.sh --uninstall llamacpp` resolves app + idempotent uninstall rc 0 (no network); `bash apps/ai/llamacpp.sh uninstall` rc 0; `make gen` idempotent; `make check` green; `make lint` 0 FAIL / 0 WARN.
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
# AI Server Start Breakage — Root Cause + Fix Design Decisions
|
||||
|
||||
**Date:** 2026-09-06
|
||||
**Architect:** Design pass over the Detective report `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (root cause classified **FACT**).
|
||||
**Constraint:** No code changes in this pass; decisions + ratified fix scope (F1-F7) for the Builder; test-fixture requirements for the Tester. Preserve the existing Bash tool architecture and the D-A..D-F stabilization decisions.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. **DQ1 — Validation stays help-gated.** The SIGPIPE race (F2) is the only validation defect; help-based validation is binary-reality-based and correct once deterministic. Version is cosmetic. "unknown" means: validate against real `--help` regardless, message without the version. NO version→capability map. [DECIDED]
|
||||
2. **DQ2 — `resolve_model` gains file-inside-dir expansion.** `$HF_DOWNLOAD_DIR/<name>`/dir → exactly-one `*.gguf` resolves to it; multiple → list + err ("pick one"); zero → err as today. Also accept `<name>/<file>.gguf`. Precedence: absolute path > dir-with-exactly-one-gguf > `$HF_DOWNLOAD_DIR/<name>.gguf` flat file > `$HF_DOWNLOAD_DIR/<name>/<file>` > relative-as-is. Existing bare-file behavior unchanged (backward compatible). [DECIDED]
|
||||
3. **DQ3 — User-bus pre-flight by default, plus `--no-unit` direct-run escape hatch (option c).** Pre-flight aborts BEFORE writing the unit (no orphan); `--no-unit` runs the server directly under nohup+pidfile for headless/SSH boxes. Shared helper `ensure_user_bus` lives in `lib/common.sh` (all three tools source it). Matrix-listener + network-download get the same pre-flight. Remediation text includes `export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`. [DECIDED]
|
||||
4. **DQ4 — `find_llamacpp` drops bare `server` and `llama.cpp/server`.** Keep `llama-server`, `llama-server-cuda`. Do NOT add `llama-server-mtl` (Mac-only; out of target). Stale-unit: start F4 pre-flight WARNS about an existing unit, does not delete it (never gratuitously remove user state). [DECIDED]
|
||||
5. **DQ5 — Always pin `--port $PORT`.** Once F2 makes validation reliable, ExecStart always carries `--port $PORT` (help-gated, so a build without `--port` omits it). No adapter probing fallback change needed (deferred). [DECIDED]
|
||||
6. **DQ6 — Installer sanity IS in scope.** `apps/ai/llamacpp.sh` post-install runs `llama-server --version` (2>&1) + `--help` and errs if the binary doesn't execute; verifies the symlink target exists. [DECIDED]
|
||||
|
||||
---
|
||||
|
||||
## DQ1: Validation posture when version is unknown
|
||||
|
||||
### Decision
|
||||
**Keep help-based validation as the single source of truth. Do NOT add a version→capability map.** "unknown" means: still validate against the real `--help` output; only the version string in log/error messages is dropped (message proceeds without version).
|
||||
|
||||
### Rationale
|
||||
Help-based validation reflects what the actual installed binary supports — it is always correct once the SIGPIPE race (F2) is fixed, because it peers at real capability output. Adding a version→capability table would introduce a second, version-coupled source of truth that drifts the moment llama.cpp adds/renames flags; its only benefit would be to decorate messages with a version string, which is cosmetic. The version is useful for *debugging* ("which build am I running") but must never gate validation. "unknown" therefore degrades only to "no version in messages", not to "no validation".
|
||||
|
||||
### How "unknown" behaves
|
||||
- `detect_llama_version` returns "unknown" only when `--version` is genuinely unreadable (no binary) — after F1, stderr is captured and the output IS readable for real builds.
|
||||
- With "unknown", `validate_*_flags` still fetch `--help` and judge flags normally; error texts simply omit the version token.
|
||||
- F1's broadened regex handles both semver and build-only strings, so "unknown" becomes rare (only truly missing binary/--version).
|
||||
|
||||
[DELIVERY: F1 capture-2>&1 + broadened regex; F2 race fix. No capability map.]
|
||||
|
||||
---
|
||||
|
||||
## DQ2: `resolve_model` directory expansion
|
||||
|
||||
### Decision
|
||||
Extend `resolve_model` (`bin/pos-ai-server:230-264`) to resolve a directory-typed explicit argument when it contains a usable model file, with exact precedence. Interactive `pick_model` stays recursive (already lists flat `.gguf` files one level up via `find -type f`; confirmed at `:210` — it lists each dir's files, so no F3 change needed there).
|
||||
|
||||
### Precedence (highest → lowest)
|
||||
1. **Absolute path** that is a file (`[ -f "$explicit" ]`) — unchanged; if the absolute path is a directory, treat as directory case below.
|
||||
2. **`$HF_DOWNLOAD_DIR/<name>` is a DIRECTORY** containing exactly one `*.gguf` → resolve to that file. If multiple `*.gguf` → print all and `err "pick one: <name>/<file>"`. If zero → fall through / err (never silently pick).
|
||||
3. **`$HF_DOWNLOAD_DIR/<name>.gguf` flat file** — unchanged (`[ -f "$candidate" ]`).
|
||||
4. **`$HF_DOWNLOAD_DIR/<name>/<file>.gguf`** (slug/file form) — resolve to `$HF_DOWNLOAD_DIR/<name>/<file>.gguf`.
|
||||
5. **Relative-as-is** — unchanged (`[ -f "$explicit" ]`).
|
||||
|
||||
### Backward compatibility
|
||||
Bare existing behavior (flat file path, absolute file path, relative path) is fully preserved — the new directory-expansion branches only fire where the current code would have errored with "Model not found". No silent picking: multiple matches always err with the disambiguating list.
|
||||
|
||||
### Exact rule (multiple-gguf dir)
|
||||
```
|
||||
if a dir contains >1 *.gguf: print each as "<dirname>/<file>" and err "model dir contains multiple — pick one"
|
||||
```
|
||||
This matches the fixture expectation (multi-gguf dir case errors with the file list).
|
||||
|
||||
[DELIVERY: F3 resolve_model expansion; no pick_model change.]
|
||||
|
||||
---
|
||||
|
||||
## DQ3: User-bus failure handling (scope across tools)
|
||||
|
||||
### Decision
|
||||
**Option (c): pre-flight err by default, `--no-unit` direct-run as escape hatch.**
|
||||
|
||||
- `ensure_user_bus` helper in **`lib/common.sh`** (the base lib already sourced by all three tools via the fallback chain; it is the natural shared home for a user-bus guard — the alterative, config-ui.sh, is a config/validation lib, not an execution helper, and would be a semantic misfit).
|
||||
- Wired into the unit-install paths of `bin/pos-ai-server:605`, `bin/pos-communication-matrix-listener:341`, and `bin/pos-network-download:191` — all three call it BEFORE `systemctl --user daemon-reload` and BEFORE the unit is written, so a failure leaves no orphaned unit.
|
||||
- On failure, `ensure_user_bus` `err`s with remediation text:
|
||||
- `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if the dir exists)
|
||||
- `sudo loginctl enable-linger $(id -un)`
|
||||
- plus the general "connect to the user's systemd bus" guidance.
|
||||
|
||||
### Why nohup/pidfile for `--no-unit` (direct-run)
|
||||
Headless/SSH boxes may lack a working user bus permanently; being unable to serve a model at all is worse than a supervisor-less process. `--no-unit` on `pos ai server start` execs `llama-server` directly under `nohup ... >$log 2>&1 &`, writes a pidfile under `$RUN_DIR`/`$HF_DOWNLOAD_DIR` sibling (e.g. `$RUN_DIR/pos-ai-server.pid`), and prints the log path + `kill $(cat pidfile)` hint. This is a thin escape hatch, NOT a second supervisor; supervision stays systemd when available.
|
||||
|
||||
### Why the helper lives in common.sh (not config-ui)
|
||||
- All three tools source `lib/common.sh` already (via the `$(dirname)/../lib/common.sh` fallback chain). Config-ui.sh is not guaranteed present in the standalone communication tools' spirit, and it is semantically a *config* lib. An *execution pre-flight* guard belongs with the other `run`/`spawn` execution helpers in common.sh.
|
||||
- Same one-line helper (declared with a `declare -F` guard like user-timers-lib.sh does) so all three tools share remediation text and behavior, and it can be unit-tested once.
|
||||
|
||||
### Minimal viable behavior for THIS pass
|
||||
- `ensure_user_bus` in common.sh.
|
||||
- Pre-flight wired into pos-ai-server, matrix-listener, network-download before unit write.
|
||||
- `--no-unit` direct-run flag on **pos-ai-server only** (the reported tool, and the only one whose unit manages llama-server). matrix-listener and network-download get the pre-flight but NO direct-run escape hatches this pass (their stateful daemons genuinely need systemd; a direct-run fallback would be a larger design change and is deferred).
|
||||
|
||||
[DELIVERY: F4 pre-flight + `--no-unit` on pos-ai-server.]
|
||||
|
||||
---
|
||||
|
||||
## DQ4: `find_llamacpp` candidates
|
||||
|
||||
### Decision
|
||||
New candidate list: **`llama-server`, `llama-server-cuda`** only. Drop bare `server` and `llama.cpp/server`. Do NOT add `llama-server-mtl`.
|
||||
|
||||
### Rationale
|
||||
- Bare `server` is an unrelated-generic-name hazard (fixture-proven: picks an unrelated binary, then ALL defaults rejected) — genuinely dangerous, drop it.
|
||||
- `llama.cpp/server` is a relative path-like token that `command -v` can only match as a literal filename `llama.cpp/server` — not a real on-PATH name for the archive's `/usr/local/bin/llama-server` layout; it provides no value, drop it.
|
||||
- `llama-server-cuda` is a genuine distinct binary name for some CUDA-series builds — keep it.
|
||||
- `llama-server-mtl` is the Apple Metal (macOS) binary name; this repo targets Debian/Ubuntu Linux only (`apps/ai/llamacpp.sh` errors on non-x64/arm64) — out of scope.
|
||||
- If `llama-server` and `llama-server-cuda` are both absent, the existing `find_llamacpp`-failure `err` already carries the installer hint (unchanged).
|
||||
|
||||
### Stale-unit handling on start (F4 interaction)
|
||||
`cmd_start` pre-flight: if `$USER_SYSTEMD_DIR/$SERVICE` already exists, **warn** that a unit is present (it may be stale/orphaned from a prior failed start) and that it will be overwritten; proceed to write. **Do NOT delete it.** `cmd_stop` (`:628-641`) is the only stated removal path and remains the exclusive way user state is removed. Rationale: deleting on start would gratuitously discard user state (a running, working unit) and would blur the unit's ownership. The pre-flight bus check (DQ3) already prevents NEW orphans, so an existing unit is either intentional or stale-legacy — warn, never auto-remove.
|
||||
|
||||
[DELIVERY: F5 candidate list + start stale-unit warning.]
|
||||
|
||||
---
|
||||
|
||||
## DQ5: Port pinning
|
||||
|
||||
### Decision
|
||||
**Always pin `--port $PORT` in ExecStart** (help-gated). Confirmed: the adapter probes and health check already use `$PORT` (default 8088; llama.cpp default is 8080 per `common/common.h:620`). After F2 makes default-flag validation reliable, `--port` is never spuriously omitted, so the server always binds the port the adapter and health check expect.
|
||||
|
||||
### Mechanics
|
||||
- `--port $PORT` is a DEFAULT flag already emitted at `bin/pos-ai-server:528` guarded by `DEFAULT_PORT_OK`. Once F2 removes the race, `DEFAULT_PORT_OK` is reliably 1 for real builds (which advertise `--port` in `--help`), so `--port $PORT` is pinned.
|
||||
- Strictly ancient llama-server builds without `--port` in `--help` omit it (help-gated); in that (degenerate) case the adapter's 8088 probe would not match the server's 8080 default. Since such builds are not the supported real release (b10822 ships `--port`, default 8080), this residual path is accepted and documented as the "ancient build" degradation.
|
||||
|
||||
### Explicitly deferred
|
||||
- **Adapter probing fallback** (probe both 8088 and 8080, or derive port from the unit): NOT in scope this pass. With F2+F5, the server and adapter agree on 8088 by construction. Adding a dual-port adapter probe would mask (not fix) a genuine port disagreement and complicate the adapter for a non-path that F5 removes. Deferred with reason.
|
||||
|
||||
[DELIVERY: F6 port pinning — achieved via F2+F5; no adapter fallback.]
|
||||
|
||||
---
|
||||
|
||||
## DQ6: Installer sanity (F7)
|
||||
|
||||
### Decision
|
||||
**In scope.** `apps/ai/llamacpp.sh:46-57` post-install adds a sanity check after the symlink loop:
|
||||
- Run `llama-server --version` capturing `2>&1` — `err` if the binary doesn't execute (non-zero exit).
|
||||
- Run `llama-server --help` similarly — `err` if unreadable.
|
||||
- Verify the symlink target exists: for each `/usr/local/bin/llama*` symlink created, `[ -e "$link" ]` (resolves target) — `err` if broken.
|
||||
|
||||
### Rationale
|
||||
This catches a genuinely broken install (missing shared lib → binary won't run; truncated/empty archive → symlink dangling) at install time with one clear `err`, instead of surfacing as a confusing "version unknown / flags rejected" on the first `pos ai server start`. It is cheap and self-contained in the installer; it is the front door to the whole tool chain and is the natural place to fail fast.
|
||||
|
||||
[DELIVERY: F7 installer post-install sanity.]
|
||||
|
||||
---
|
||||
|
||||
## Ratified Fix Scope (F1-F7)
|
||||
|
||||
| # | File:line | Decision | 1-line change |
|
||||
|---|-----------|----------|---------------|
|
||||
| **F1** | `bin/pos-ai-server:71` | **KEEP** | Capture `2>&1` and broaden regex to `[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+`. |
|
||||
| **F2** | `bin/pos-ai-server:101,145` | **KEEP** | Replace `printf|grep -q` pipeline with `grep -E -- … >/dev/null <<<"$help_text"` (non-q, no pipe → no SIGPIPE); apply identically to `validate_requested_flags` and `validate_default_flags`. |
|
||||
| **F3** | `bin/pos-ai-server:230-264` | **KEEP** | `resolve_model` dir-expansion per DQ2 precedence (exactly-one-gguf → resolve; multiple → list+err; commit `$HF_DOWNLOAD_DIR/<name>/<file>` form). |
|
||||
| **F4** | `bin/pos-ai-server:605` + `:461-626`; `bin/pos-communication-matrix-listener:341`; `bin/pos-network-download:191`; **new** `lib/common.sh` | **KEEP (adjusted)** | Add `ensure_user_bus` to `lib/common.sh`; call before unit write in all three tools (abort, no orphan); plus `--no-unit` direct-run flag on `pos-ai-server`; start pre-flight warns on existing unit (DQ4). |
|
||||
| **F5** | `bin/pos-ai-server:56` | **KEEP** | `find_llamacpp` candidates → `("llama-server" "llama-server-cuda")`. |
|
||||
| **F6** | `bin/pos-ai-server:528` (+ no adapter change) | **KEEP** | Confirm `--port $PORT` always pinned (reliably emitted once F2 fixed); no adapter probing fallback. |
|
||||
| **F7** | `apps/ai/llamacpp.sh:46-57` | **KEEP (in scope)** | Post-install sanity: `llama-server --version` (2>&1) + `--help` execute, symlink targets exist, else `err`. |
|
||||
|
||||
## Explicitly deferred (NOT this pass)
|
||||
|
||||
1. **Adapter probing fallback** (probe 8088+8080 / derive port from unit) — F6 makes the adapter agree with the server by construction; a dual-port probe would mask (not fix) disagreement. Deferred with reason.
|
||||
2. **Direct-run escape hatch for matrix-listener and network-download** — stateful daemons genuinely need systemd; a direct-run supervisor is a larger design change. Deferred; only pos-ai-server gets `--no-unit` this pass.
|
||||
3. **Multi-user / TELEGRAM_GROUP_MODE and DQ-adjacent chat features** — unrelated to this breakage; tracked separately in the stabilization design.
|
||||
4. **D1 (pos-ai-hf single-file download failure recording)** and **LLAMACPP_HOST coherence (D2)** — separate defects already tracked in the stabilization open-items; not part of this fix map.
|
||||
5. **`--no-mmap`-style substring false positives beyond the current word-boundary regex** — D4 (already implemented in this file at `:101`); no change needed unless F2 rewrites it, which uses the identical word-boundary pattern.
|
||||
|
||||
---
|
||||
|
||||
## Test-Fixture Requirements (for Tester)
|
||||
|
||||
Per fix from the Detective's fixture spec, deterministically:
|
||||
|
||||
- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit …)` **to stderr**, help to stdout → assert `detect_llama_version` returns `0.4.0` (not "unknown"). Also a fixture printing only `version: b10822`/`build 10822` to stderr → assert broadened regex returns the build token.
|
||||
- **F2 (race + determinism):** fake `llama-server` whose `--help` emits a 59 KB body with `--threads`@line 7, `--ctx-size`@25, `--n-gpu-layers`@140, `--host`/`--port` near end (real llama.cpp layout). Run `validate_default_flags` **N≥20 times under `set -euo pipefail`** → fixed code reports all 5 supported on EVERY run (zero omissions). Regression tail: pre-fix code must fail at least once in the same loop (proves the race existed).
|
||||
- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `DRY_RUN=1 pos ai server start Qwen-Qwen3-1.7B-GGUF` resolves to the file; multi-gguf dir case errors listing each as `dirname/file`; zero-gguf dir errors as today.
|
||||
- **F4 (bus):** env with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or stub `systemctl` failing with the dbus message) → assert pre-flight `err`s with remediation text (both `export XDG_RUNTIME_DIR=…` and `sudo loginctl enable-linger …`), and **no unit file is written** (no orphan). Assert `--no-unit` path execs the direct command via a stub `llama-server` recording argv + writes pidfile.
|
||||
- **F5 (server fallback):** PATH containing ONLY an unrelated fake `server` (its own --help/--version), no llama-server → assert `find_llamacpp` does NOT return `server`.
|
||||
- **F6 (port):** with F2 fixed, `DRY_RUN=1` generate a unit → assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`).
|
||||
- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (missing shared lib) → assert post-install sanity `err`s.
|
||||
- **E2E regression (reported user scenario):** with stubs (fake `llama-server` b10822-shaped stderr + help, fake systemctl that succeeds only with `XDG_RUNTIME_DIR` set), run install→`pos ai server start Qwen-Qwen3-1.7B-GGUF`→assert the written unit is correct (ExecStart carries all 5 flags + `--port 8088`, no orphan unit, version resolvable).
|
||||
|
||||
---
|
||||
|
||||
*Deliverables complete. See `AgentsReport/architect/2026-09-06_stabilization-design.md` (appended post-merge section) for the consolidated record.*
|
||||
@@ -768,3 +768,15 @@ Groups A, B, C, and D are **parallel-safe** (no file overlap). Group E depends o
|
||||
4. **pos-ai-hf single-file download failure (D1 from AI audit):** The single-file download path does not record failures in `failed_files`, so `.hf-meta` is written for a partially-downloaded model. This is a HIGH defect but is a pure bug fix (not an architectural decision) — belongs in Builder scope directly, not in this design pass.
|
||||
5. **GPG passphrase in argv (V3 from Security audit):** Fixing this requires `gpg --batch --passphrase-fd` or a temp-file approach. Behavioral change to `bin/pos-system-backup` — defer to a separate fix commit.
|
||||
6. **H-002/H-003 bashrc/bash_completion over-broad removal:** Partially addressed in D-E (`.bash_completion` fix) but a more thorough marker-based approach would be ideal. The D-E fix is the minimum viable improvement.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-06 (post-merge) — llama-server start breakage root cause + fix decisions
|
||||
|
||||
**References:** `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (root cause, classified **FACT**); full decisions in `AgentsReport/architect/2026-09-06_ai-server-fix-design.md`.
|
||||
|
||||
**Summary:** A user's `pos ai server start` failing after `bash apps/install.sh llamacpp` traced to four independent defects in `bin/pos-ai-server` (plus one adapter-default interaction): (1) `detect_llama_version:71` discards stderr, and llama.cpp b10822 prints its version to stderr → always "unknown"; (2) `printf|grep -q` under `set -o pipefail` → SIGPIPE race (rc=141) → flags randomly rejected; (3) `resolve_model:229-259` rejects the HF-downloader's `<dir>/<file>.gguf` layout; (4) unit write precedes an unguarded `systemctl --user daemon-reload:605` that fails under SSH w/o `XDG_RUNTIME_DIR` → orphaned unit. Port pinning (llama.cpp default 8080 vs tool/adapter 8088) is downstream of the race.
|
||||
|
||||
**Decisions added (DQ1-DQ6, see the fix-design file for full rationale):** help-gated validation retained (no version→capability map; version is cosmetic); `resolve_model` expands a single-gguf dir (exact precedence, no silent multi-pick); user-bus pre-flight via new `ensure_user_bus` in `lib/common.sh` + `--no-unit` direct-run on `pos-ai-server`; `find_llamacpp` narrowed to `llama-server`/`llama-server-cuda`; `--port $PORT` always pinned; installer post-install sanity in scope. **Ratified fix scope F1-F7, all KEEP** (F4 adjusted to add `--no-unit` + shared helper; F2's SIGPIPE mechanism supersedes the earlier D-F's descriptive reliance on the broken pipeline). **Deferred:** adapter dual-port probing fallback, direct-run for matrix-listener/network-download, and the previously-tracked D1/D2 AI-server defects.
|
||||
|
||||
**Consistency with prior decisions (no D-A..D-F text altered above):** D-F's requested-vs-default validation semantics (requested → hard err, default → warn+omit) remains the contract; F2 fixes the mechanism that made D-F's help-gate non-deterministic, and F6 (port pinning) is a direct realization of D-F's always-emitted-defaults guarantee. The bus pre-flight (F4) extends what `scheduler-lib.sh:614` already does (a *query* guard) into a hard pre-write *enable* guard shared across the three tools that write user units.
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Builder Report — llama-server breakage fixes (F1-F7)
|
||||
|
||||
**Date:** 2026-09-06
|
||||
**Builder:** Implementation of Architect DQ1-DQ6 ratified scope F1-F7.
|
||||
**Bindings:** `AgentsReport/architect/2026-09-06_ai-server-fix-design.md` (FACT), `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (FACT root-cause evidence).
|
||||
|
||||
## TL;DR
|
||||
|
||||
Status: **IMPLEMENTED** (all F1-F7)
|
||||
|
||||
Files changed:
|
||||
- `bin/pos-ai-server` (F1, F2, F3, F4, F5, F6)
|
||||
- `lib/common.sh` (F4 — new `ensure_user_bus`)
|
||||
- `bin/pos-communication-matrix-listener` (F4 — inline + call)
|
||||
- `bin/pos-network-download` (F4 — call)
|
||||
- `apps/ai/llamacpp.sh` (F7 — post-install sanity)
|
||||
- `DOC/POS.md` (minimal — `--no-unit` documented)
|
||||
- generated docs via `make gen` (`DOC/AGENT_Context_Project.md`, `completions/pos.bash`)
|
||||
|
||||
Verification: `bash -n` all touched; probes 1-8 = **63/63 PASS, rc=141 count 0**; `make gen` idempotent x2; `make check` green; `make lint` `0 FAIL, 0 WARN`.
|
||||
|
||||
Probe workspace: `/tmp/opencode/ai-probes/` (stubs, fixtures, `run-probes.sh`, `PROBES-RESULT.txt`).
|
||||
|
||||
---
|
||||
|
||||
## Step 1: F1 — `detect_llama_version` stderr capture + broadened regex
|
||||
|
||||
`bin/pos-ai-server:82-90`. Real llama.cpp prints `version: 0.4.0-dev (build 10822, commit …)` to **STDERR**; the old code captured only stdout and regexed `[0-9]+\.[0-9]+\.[0-9]+` alone.
|
||||
|
||||
- Both streams captured: `"$bin" --version 2>&1`.
|
||||
- Regex accepts `X.Y.Z` OR `build [0-9]+` OR `b[0-9]+` (old builds).
|
||||
- `"${version#build }"` strips the phrase so `build 10822` displays as `10822`.
|
||||
- Guarded: missing binary / unreadable output → `unknown` (no errexit).
|
||||
|
||||
Evidence (P1): stderr semver → `0.4.0`; `build 10822` (stderr) → `10822`; `b10822` → `b10822`; missing binary → `unknown`. [DONE]
|
||||
|
||||
## Step 2: F2 — pipe-less flag validation (no rc=141 race)
|
||||
|
||||
`bin/pos-ai-server:97-124` (`validate_requested_flags`), `:135-170` (`validate_default_flags`). Old form `printf '%s' "$help_text" | grep -qE ...` died of SIGPIPE (rc=141) when the >64KB pipe-adjacent grep exited at the first match — valid flags sporadically judged unsupported. New form: non-quiet `grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null` (herestring needs no pipe; non-q grep consumes the whole input; same word-boundary regex and guard shape).
|
||||
|
||||
Evidence (P2): on the 58,362-byte/732-line fixture help, 25 runs of `validate_default_flags` → all 5 defaults kept every run, `rc=141` count **0**; requested-flag path: supported `--port` rc 0, unsupported flag rc 1 with version-aware err. [DONE]
|
||||
|
||||
## Step 3: F3 — `resolve_model` directory expansion (never silently pick)
|
||||
|
||||
`bin/pos-ai-server:213-264` + helper `resolve_gguf_in_dir` (`:196-210`). DQ2 precedence: absolute file → absolute dir with exactly one `.gguf` → `$HF_DOWNLOAD_DIR/<name>/<file>.gguf` (slug/file form) → `$HF_DOWNLOAD_DIR/<name>.gguf` (flat file) → relative-as-is. A dir with multiple `.gguf`s errors listing each as `<dirname>/<file>` + "pick one"; zero `.gguf`s falls through to the today's `Model not found:` err.
|
||||
|
||||
Evidence (P3): single-gguf dir resolves; slug/file form resolves; multi-gguf errors listing both files + "pick one"; empty dir → rc 1 with today's `Model not found: empty-dir` text. [DONE]
|
||||
|
||||
## Step 4: F4 — `ensure_user_bus` + `--no-unit` escape hatch
|
||||
|
||||
`lib/common.sh` (new `ensure_user_bus`, after `confirm()`): `systemctl --user show-environment` reachability probe; on failure `err` with BOTH remediation lines (`export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`) + "no unit was written". Called:
|
||||
|
||||
- `bin/pos-ai-server` `cmd_start` (before DRY_RUN return; only when `--no-unit` not given). `--no-unit` flag: added to `# POS_FLAGS:`, parse case, usage; direct run via `eval "nohup $exec_cmd >'$NO_UNIT_LOG' 2>&1 &"`, pidfile/log under `${XDG_RUNTIME_DIR:-/tmp}`; `cmd_stop`/`cmd_status` pidfile-aware (`kill -0`, `^[0-9]+$` guard).
|
||||
- `bin/pos-communication-matrix-listener` — **factually does NOT source `lib/common.sh`** (inline `err`/`log`/`warn` at lines 19-21); got an inline `declare -F ensure_user_bus`-guarded duplicate + call in `enable_service` (architect's "all three source common.sh" premise was wrong; sourcing it would redefine helpers with colors and change output).
|
||||
- `bin/pos-network-download` `cmd_start` else-branch (first statement before unit write).
|
||||
|
||||
Evidence (P4): with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset and a failing `systemctl` stub: pos-ai-server DRY_RUN start rc 1 with both remediation lines and **no unit written**; matrix-listener `--enable` rc 1, no unit; network-download `start` rc 1, no unit. [DONE]
|
||||
|
||||
## Step 5: F5 — `find_llamacpp` candidate narrowing
|
||||
|
||||
`bin/pos-ai-server:56-61`. Candidates narrowed to `("llama-server" "llama-server-cuda")` (dropped bare `server` and `llama.cpp/server` which had previously matched unrelated binaries). Stale-unit overwrite warn added in start; user unit state never deleted.
|
||||
|
||||
Evidence (P5): PATH containing only an unrelated binary named `server` → `find_llamacpp` returns nothing, rc 1; stub `llama-server` → rc 0, returns `llama-server`. [DONE]
|
||||
|
||||
## Step 6: F6 — port pinning preserved (no change needed, verified)
|
||||
|
||||
Verified via P6: DRY_RUN start with the b10822-shaped stub help → ExecStart contains all 5 defaults (`--host 127.0.0.1`, `--port 8088`, `--n-gpu-layers`, `--ctx-size 4096`, `--threads`) and the version resolves to `0.4.0` (not `unknown`). [DONE]
|
||||
|
||||
## Step 7: F7 — llamacpp.sh post-install sanity
|
||||
|
||||
`apps/ai/llamacpp.sh`: new `llamacpp_sanity()` (seam `LLAMACPP_BIN_DIR="${LLAMACPP_BIN_DIR:-/usr/local/bin}"`), called after install `spawn`. Verifies: `[ -e "$bin" ]` (catches dangling symlink), `command -v llama-server`, `llama-server --version >/dev/null 2>&1`, `llama-server --help >/dev/null 2>&1`; on success logs `llama.cpp sanity OK`.
|
||||
|
||||
Evidence (P7): normal fixture → rc 0 + OK log; dangling-symlink fixture → rc 1 with "dangling symlink"; missing-shared-lib fixture → rc 1 with the binary's stderr text. [DONE]
|
||||
|
||||
## Step 8: E2E user scenario
|
||||
|
||||
Evidence (P8): positive (bus env set, `systemctl` stub succeeds): `pos ai-server start Qwen-Qwen3-1.7B-GGUF` rc 0, unit written with `--port 8088`, all 4 other defaults, model path expanded from dir, **no "unknown" version anywhere in start output**. SSH-shaped (no bus env, failing stub): rc 1 with both remediation lines, **no orphan unit**. [DONE]
|
||||
|
||||
## Gates
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| `bash -n` on 5 touched scripts | pass (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh) |
|
||||
| `make gen` x2 | rc 0 both; `git diff` identical → **idempotent** |
|
||||
| `make check` | `check-sync: OK` |
|
||||
| `make lint` | `0 FAIL, 0 WARN (convention lint)` |
|
||||
| Probes 1-8 | **63 PASS / 0 FAIL / rc=141 count 0** |
|
||||
|
||||
## Scope compliance
|
||||
|
||||
- Touched exactly the in-scope files (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh, DOC/POS.md, gen output). `tests/`, `scripts/lint-conventions.sh`, `bin/pos-system-uninstall`, `lib/config-ui.sh` untouched.
|
||||
- Unrelated pre-existing dirty file `AgentsReport/architect/2026-09-06_stabilization-design.md` NOT modified by Builder.
|
||||
- No commits made.
|
||||
|
||||
## Residual risks / known deviations
|
||||
|
||||
1. `--no-unit` direct run uses `eval "nohup $exec_cmd … &"` — tool-generated string; only binary (validated by `command -v`) and model (validated path) tokens are interpolated, both quoted; matches the namespace's bash DNA.
|
||||
2. F1 displays `build 10822` as `10822` via `"${version#build }"` — slight deviation from the architect's literal regex output wording, required to match probe 1's `10822` expectation; behavior documented at `bin/pos-ai-server:75-81`.
|
||||
3. `matrix-listener` carries an inline duplicate of `ensure_user_bus` (guarded by `declare -F` so the shared one wins if ever sourced) — the file deliberately does not source common.sh (would change helper output). If a future task refactors matrix-listener to source common.sh, the inline copy can be deleted.
|
||||
4. E2E positive run writes/overwrites a unit under `$USER_SYSTEMD_DIR` only inside the probe workspace — no real system units touched, no processes spawned (stubs only).
|
||||
5. Runtime flags validated by grep use `>`-redirected non-quiet grep instead of `grep -q`: a 58KB fixture costs ~1-2ms per flag — negligible for a unit-write path.
|
||||
|
||||
Recommended next agent: **Reviewer** (independent adversarial review of the F1-F7 diff before acceptance).
|
||||
@@ -0,0 +1,56 @@
|
||||
# Builder report — F1 regex fix in bin/pos-ai-server
|
||||
|
||||
Date: 2026-09-06 · Builder · Objective: fix `detect_llama_version` regex so `build 1.2.3` returns `1.2.3` not `1`. File scope: `bin/pos-ai-server` ONLY. No commit.
|
||||
|
||||
## TL;DR (status: IMPLEMENTED)
|
||||
|
||||
- **Root cause:** `grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+'` picks the LEFTMOST match; for input `build 1.2.3`, `build [0-9]+` wins → `build 1` → after `${version#build }` → `1`.
|
||||
- **Fix:** `bin/pos-ai-server:86` → `build [0-9]+(\.[0-9]+)*` (full dotted tail consumed). Stripping and guarded behavior unchanged.
|
||||
- **Gates:** `bash -n` clean; `make gen` rc 0 with generated files byte-identical (sha256 before == after); `make check` OK; `make lint` 0 FAIL / 0 WARN.
|
||||
- **Tests:** `tests/t-ai-llama-detect.sh` (was failing on F1) now 9/9 PASS; `tests/t-ai-server-validate.sh` (Tester's F1 suite, real extracted function) 27/27 PASS.
|
||||
|
||||
## Step 1: Read & confirm scope
|
||||
- Read `AgentsReport/tester/2026-09-06_ai-server-tests.md` (F1 finding: lines 9, 63).
|
||||
- Read `bin/pos-ai-server` (lines 82–90).
|
||||
- Scope confirmed: touch `bin/pos-ai-server` only; do NOT touch tests/.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 2: Apply regex fix
|
||||
- `bin/pos-ai-server:86`: `build [0-9]+` → `build [0-9]+(\.[0-9]+)*`.
|
||||
- Stripping (`version="${version#build }"`) and guarded behavior (`command -v` guard + `|| true` + `unknown` fallback) unchanged — per brief.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 3: Verification matrix (shipped regex evaluated against stub binaries)
|
||||
- `version: 0.4.0-dev (build 10822, commit …)` (stderr) → `0.4.0` PASS
|
||||
- `build 10822` → `10822` PASS
|
||||
- `build 1.2.3` → `1.2.3` PASS (regression fixed)
|
||||
- `b10822` → `b10822` — see Decision Note below (brief's conditional `10822` not taken)
|
||||
- missing binary → `unknown` PASS
|
||||
- unreadable / non-matching output → `unknown` PASS
|
||||
- Additional: `semver only (1.5.0)` → `1.5.0` PASS (leftmost semver still wins)
|
||||
|
||||
### Decision Note (b-token)
|
||||
Brief lists `b10822` → `10822` "if your regex keeps a b-token", but also mandates "keep the same stripping logic" (`${version#build }` — strips only `build `, never `b`). The two are mutually exclusive. Resolution per the hard constraint: kept the b-token (matches existing comment: very old builds print no semver) and kept stripping identical → `b10822` yields `b10822`. This is exactly what the Tester's own fixture asserts (`tests/t-ai-server-validate.sh:118` → `b10822`), and `t-ai-server-validate.sh` passes 27/27 with the real extracted function.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 4: Gates
|
||||
- `bash -n bin/pos-ai-server` → clean
|
||||
- `make gen` → rc 0; sha256 of `DOC/AGENT_Context_Project.md` + `completions/pos.bash` identical before/after → NO output change (regex is code, not a GEN: block)
|
||||
- `make check` → OK
|
||||
- `make lint` → 0 FAIL, 0 WARN
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 5: Tester-requested regression verification
|
||||
- `./tests/run-tests.sh t-ai-llama-detect.sh` → PASS, 9/9 checks (was the failing test per Tester F1 finding; `llama.cpp build 1.2.3` fixture now yields `1.2.3`).
|
||||
- `./tests/run-tests.sh t-ai-server-validate.sh` → PASS, 27/27 checks (real extracted `detect_llama_version`, incl. `build 10822` → `10822` and `b10822` → `b10822` fixtures).
|
||||
|
||||
[PASS]
|
||||
|
||||
## Handoff
|
||||
- Status: IMPLEMENTED. Files changed: `bin/pos-ai-server` (single regex line, line 86). Git: NOT committed (per brief).
|
||||
- Note: the working tree already carried the pre-existing uncommitted F1–F7 Builder changes plus 4 untracked Tester test files (per Tester report Step 6); none of that is mine.
|
||||
- Recommended next: Reviewer (independent gate + `make test` re-run). Separate track: F4 bus-gap fixture updates in 3 pre-existing DRY_RUN tests are already routed to Orchestrator — not touched here.
|
||||
@@ -0,0 +1,282 @@
|
||||
# Detective Report: `pos ai server start` — llama.cpp Post-Install Breakage
|
||||
|
||||
**Date:** 2026-09-06
|
||||
**Investigator:** Detective (read-only)
|
||||
**Symptom (user paste):** After `bash apps/install.sh llamacpp` installed llama.cpp b10822, `pos ai server start Qwen-Qwen3-1.7B-GGUF` fails: version "unknown", default flags rejected, model-not-found, and dbus error at the end.
|
||||
**Severity:** High — complete server-start failure after a clean install; the generated unit is silently corrupted.
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Hyp | Verdict | One-line evidence |
|
||||
|-----|---------|-------------------|
|
||||
| H1 | **CONFIRMED** (wrong mechanism) | `llama-server --version` prints to **stderr** (`common/build-info.h:13` default `FILE* = stderr`); `detect_llama_version` discards it with `2>/dev/null` → always "unknown" |
|
||||
| H2 | **REFUTED as user's cause; latent defect stands** | Archive ships `llama-server`; installer symlinks it (verified); but bare `server` fallback picks unrelated binaries when llama-server is missing (fixture-proven) |
|
||||
| H3 | **REFUTED as stated; REAL bug found** | Word-boundary regex MATCHES all 5 flags in real help; the actual failure is `printf|grep -q` + `set -o pipefail` **SIGPIPE race** (pipeline rc=141) — flaky per-run |
|
||||
| H4 | **CONFIRMED** | `resolve_model` accepts only files; a dir containing one `.gguf` under `HF_DOWNLOAD_DIR` fails with exact user error (reproduced) |
|
||||
| H5 | **CONFIRMED** | Unit written (`:584-602`) BEFORE `systemctl --user daemon-reload` (`:605`); no bus pre-check; SSH without `XDG_RUNTIME_DIR` → exact error, `set -e` aborts, linger hint never runs; orphaned unit remains |
|
||||
| H6 | **CONFIRMED** | llama.cpp default port **8080** (`common/common.h:620`; `--help` default), tool/adapter default **8088**; real orphaned unit on this machine omits `--port` |
|
||||
|
||||
**Root cause chain (one paragraph):** `detect_llama_version` (bin/pos-ai-server:71) always returns "unknown" because llama.cpp's `--version` writes to stderr and the tool discards stderr (`2>/dev/null`). Independently, `validate_default_flags`/`validate_requested_flags` (lines 101, 145) run `printf '%s' "$help_text" | grep -qE ...` under `set -o pipefail`; `grep -q` exits at the first match (flags are at byte offsets 320–39,741 of a 59,000-byte help), the bash-builtin `printf` then hits EPIPE, and pipefail promotes the SIGPIPE (rc=141) — so the `if` is false even though the flag IS in the help. Which flags "fail" each run is a scheduling race (empirically 0–4 flags rejected per run). Separately, `resolve_model` (lines 229-259) rejects the user's directory argument `Qwen-Qwen3-1.7B-GGUF` (HF downloader produces `$HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf`, not a flat file), and the unit write precedes an unguarded `systemctl --user daemon-reload` that fails under SSH with no `XDG_RUNTIME_DIR`, aborting via `set -e` before the linger hint and leaving an orphaned unit whose ExecStart may already have lost `--port` (real example on this machine: `ExecStart=... --host 127.0.0.1` only), guaranteeing a port mismatch against the adapter's 8088 health/API probes.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: H1 — Version detection "unknown"
|
||||
|
||||
### Hypothesis (as briefed)
|
||||
`detect_llama_version` (`bin/pos-ai-server:65-74`) greps `[0-9]+\.[0-9]+\.[0-9]+` (semver) from `--version`; llama.cpp uses build numbers `bNNNNN`, never X.Y.Z.
|
||||
|
||||
### Actual llama.cpp `--version` output (b10822)
|
||||
Downloaded and executed the real release binary (`llama-b10822-bin-ubuntu-x64.tar.gz`, `https://github.com/ggml-org/llama.cpp/releases/tag/b10822`):
|
||||
|
||||
```
|
||||
$ ./llama-server --version
|
||||
version: 0.4.0-dev (build 10822, commit c457e3bf7)
|
||||
built with GNU 11.4.0 for Linux x86_64
|
||||
```
|
||||
|
||||
**The format DOES contain X.Y.Z (`0.4.0`).** The briefed premise is factually wrong for current builds. The regex would match `0.4.0`.
|
||||
|
||||
### The REAL mechanism — output goes to stderr
|
||||
```
|
||||
$ ./llama-server --version 1>/dev/null # output STILL appears
|
||||
$ ./llama-server --version 2>/dev/null # NOTHING appears
|
||||
```
|
||||
`od -c` confirms the bytes are written to **stderr**. Source citation: `common/build-info.h:13`
|
||||
```cpp
|
||||
void llama_print_build_info(const char *, FILE * = stderr);
|
||||
```
|
||||
called from `common/arg.cpp:1456` (`llama_print_build_info(llama_version()); exit(0);`) — default stream stderr.
|
||||
|
||||
`detect_llama_version` (`bin/pos-ai-server:71`):
|
||||
```bash
|
||||
version="$("$bin" --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
|
||||
```
|
||||
`2>/dev/null` discards the ONLY output → grep gets empty input → `version=` empty → `"unknown"`. Proven against real binary: with `2>/dev/null` captured nothing; with `2>&1` captured `0.4.0`.
|
||||
|
||||
Note: even with `2>&1`, the regex is brittle against older builds that print only `build 10822` (no semver) — a secondary hazard, not the current blocker.
|
||||
|
||||
```
|
||||
Hypothesis: version always "unknown" because regex can't match
|
||||
Why plausible: builds are tag "b10822"
|
||||
Evidence supporting: user sees "unknown" in all messages
|
||||
Evidence against: actual output has X.Y.Z ("0.4.0"); the regex DOES match
|
||||
Test needed: run detect_llama_version verbatim against real binary
|
||||
Result: empty capture (stderr discarded) → "unknown"
|
||||
Conclusion: CONFIRMED outcome, wrong mechanism — stderr discard, not regex format
|
||||
```
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Step 2: H2 — Wrong binary found
|
||||
|
||||
### (a) What the release archive ships
|
||||
`llama-b10822-bin-ubuntu-x64.tar.gz` (downloaded and listed) contains `llama-server` (among ~30 `llama*` tools + `.so` libs). **No bare `server` binary.**
|
||||
|
||||
### (b) Installer symlink behavior
|
||||
`apps/ai/llamacpp.sh:52-55`:
|
||||
```bash
|
||||
for bin in \$install_dir/llama*; do
|
||||
[ -f "\$bin" ] && [ -x "\$bin" ] || continue
|
||||
sudo ln -sf "\$bin" /usr/local/bin/\$(basename "\$bin")
|
||||
done
|
||||
```
|
||||
Since the archive ships `llama-server`, the glob `llama*` matches it → `/usr/local/bin/llama-server` IS created. Verified on the target machine (real install): `/usr/local/bin/llama-server -> /usr/local/lib/llama.cpp-b10822/llama-server`, and `llama-server --help` (PATH lookup) returns the full 732-line/59,000-byte help. RUNPATH `$ORIGIN` (`readelf -d`: `Library runpath: [$ORIGIN]`) makes the shared libs resolve through the symlink. **The install is not broken this way — H2 is not the user's cause.**
|
||||
|
||||
### (c) The bare `server` fallback is a genuine latent defect (fixture)
|
||||
`find_llamacpp` (`bin/pos-ai-server:56`) candidates: `llama-server`, `llama.cpp/server`, `server`, `llama-server-cuda`. Fixture: PATH containing ONLY an unrelated `/tmp/h2fixture/bin2/server` (no llama-server):
|
||||
|
||||
```
|
||||
find_llamacpp -> 'server' (command -v -> '/tmp/h2fixture/bin2/server')
|
||||
detect_llama_version -> (depends on the unrelated binary)
|
||||
validate_default_flags:
|
||||
--port: UNSUPPORTED (warn + omit)
|
||||
--host: UNSUPPORTED (warn + omit)
|
||||
--n-gpu-layers: UNSUPPORTED (warn + omit)
|
||||
--ctx-size: UNSUPPORTED (warn + omit)
|
||||
--threads: UNSUPPORTED (warn + omit)
|
||||
```
|
||||
i.e., a wrong-binary scenario produces the same family of user-visible messages — but **not** what this user hit (their install is correct and `llama-server` resolves first because candidate order checks `llama-server` before `server`).
|
||||
|
||||
```
|
||||
Conclusion: REFUTED as user's root cause; PARTIALLY-CONFIRMED as latent defect (bare `server` fallback)
|
||||
```
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Step 3: H3 — Help-format regex vs real llama-server --help — **actual root cause of the flag rejections**
|
||||
|
||||
### The regex matches real help
|
||||
Real `llama-server --help` (732 lines) contains (`grep -n` on real output):
|
||||
```
|
||||
7: -t, --threads N
|
||||
25: -c, --ctx-size N
|
||||
140: -ngl, --gpu-layers, --n-gpu-layers N
|
||||
503: --host HOST
|
||||
506: --port PORT
|
||||
```
|
||||
Word-boundary regex `(^|[[:space:]])${flag}([[:space:]]|=|$)` matches all five when tested WITHOUT pipefail (verified repeatedly). H3 as stated ("format mismatch misses real flags") is **REFUTED**.
|
||||
|
||||
### The real failure: `printf | grep -q` + `set -o pipefail` → SIGPIPE race
|
||||
`bin/pos-ai-server` line 2: `set -euo pipefail`. Lines 101/145:
|
||||
```bash
|
||||
if printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then
|
||||
```
|
||||
`grep -q` exits as soon as it finds a match (closing the pipe's read end). The produced help_text is 59,000 bytes; the pipe buffer is 64 KB. Flags appear at byte offsets:
|
||||
- `--threads` 320, `--ctx-size` 1892, `--n-gpu-layers` 10992 (early)
|
||||
- `--host` 39499, `--port` 39741 (late)
|
||||
|
||||
For early flags, grep matches and exits after reading ≤ a few KB; the bash-builtin `printf` still has ~57 KB to write → EPIPE → under `pipefail` the pipeline returns **141 (128+13=SIGPIPE)** → the `if` is false even though grep found the flag. For late flags, printf usually completes writing into the 64 KB buffer before grep exits → rc=0. It's a **race**, so outcomes vary run to run.
|
||||
|
||||
**Isolated proof** (same capture, same regex, pipefail on):
|
||||
```
|
||||
++ printf '%s' "$help_text" | grep -qE -- '(^|[[:space:]])--ctx-size([[:space:]]|=|$)'
|
||||
pipeline rc=141 (grep -q DID match; printf died of SIGPIPE)
|
||||
```
|
||||
30-run trial: MATCH/NOMATCH alternated ~50/50 for `--ctx-size`.
|
||||
|
||||
**Live `pos-ai-server` flakiness (10 runs, real binary, identical inputs):**
|
||||
```
|
||||
run 1: --ctx-size --threads
|
||||
run 2: --ctx-size --threads
|
||||
run 3: --threads
|
||||
run 4: --threads
|
||||
run 5: --n-gpu-layers --ctx-size --threads
|
||||
run 6: --ctx-size
|
||||
run 7: --port --n-gpu-layers --threads
|
||||
run 8: --host --n-gpu-layers --ctx-size --threads
|
||||
run 9: --n-gpu-layers --ctx-size --threads
|
||||
run 10: (none!)
|
||||
```
|
||||
This explains the user's per-run differences (runs 1-2 vs run 3): **there is no CLI/config/env difference** — `REQUESTED_FLAGS` and `CONFIG_REQUESTED_FLAGS` are empty (no ai.env `LLAMACPP_*` keys, no CLI flags; verified the `requested_from_env_config` calls return early). Flag validation runs identically each time; the outcome is a scheduling race. Both `validate_default_flags` (warn+omit) and `validate_requested_flags` (hard `err`) carry the same bug — a requested `--ctx-size` would randomly hard-fail with "does not expose".
|
||||
|
||||
Verdict: **H3 REFUTED as stated; SIGPIPE+pipefail race is THE root cause of the flag rejections.**
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Step 4: H4 — Model resolution
|
||||
|
||||
`resolve_model` (`bin/pos-ai-server:229-259`) accepts: absolute FILE path (`[ -f ]`), `$HF_DOWNLOAD_DIR/<name>` **FILE** (`[ -f "$candidate" ]`), or relative FILE. It never treats a DIRECTORY under `$HF_DOWNLOAD_DIR` as a model.
|
||||
|
||||
Downloader layout (`bin/pos-ai-hf`): `hf_repo_dir()` (lines 308-311) → `$HF_DOWNLOAD_DIR/${repo_id//\//-}` — e.g. repo `Qwen/Qwen-Qwen3-1.7B-GGUF` → dir `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/` containing `Qwen3-1.7B-Q8_0.gguf` + `.hf-meta`.
|
||||
|
||||
Reproduction (fixture dir created exactly like the user's; `.gguf` stub inside):
|
||||
```
|
||||
$ pos-ai-server start Qwen-Qwen3-1.7B-GGUF
|
||||
[!] installed llama.cpp unknown does not support default flag --threads — omitting it from the unit
|
||||
ERROR: Model not found: Qwen-Qwen3-1.7B-GGUF (also searched /home/unknown/.local/share/linux_post_install/ai/models)
|
||||
EXIT: 1
|
||||
```
|
||||
Exact user message. The user's `mv Qwen3-1.7B-Q8_0.gguf ../` moved the file into `$HF_DOWNLOAD_DIR/`, but the arg `Qwen-Qwen3-1.7B-GGUF` is still a dir name → still fails (run 2 reproduced identically). The absolute file path (run 3) succeeds at model resolution (reproduced).
|
||||
|
||||
`pick_model` (lines 205-227) uses `find "$HF_DOWNLOAD_DIR" -name '*.gguf' -type f` — RECURSIVE, does NOT miss subdirs (verified). But it is only reached when no explicit arg/config is given; the user passed an explicit arg, so it wasn't involved.
|
||||
|
||||
**H4 CONFIRMED.**
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Step 5: H5 — dbus/systemctl under SSH
|
||||
|
||||
Code ordering (`bin/pos-ai-server`):
|
||||
- `set -euo pipefail` (line 2)
|
||||
- unit write + `chmod 644` (lines 584-602)
|
||||
- `systemctl --user daemon-reload` (line 605)
|
||||
- `systemctl --user enable --now "$SERVICE"` (line 606)
|
||||
- linger hint (lines 610-615)
|
||||
|
||||
No pre-check of the user bus anywhere in the file. Same pattern in `pos-communication-matrix-listener:341-342` and `pos-network-download:191-192`; NO tool pre-checks the bus (`pos-entertainment-status:55` uses `systemctl --user show-environment` only as a query guard, not an enable guard).
|
||||
|
||||
Reproduced exactly on target machine (SSH-like shell: `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS` unset):
|
||||
```
|
||||
$ systemctl --user show-environment
|
||||
Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)
|
||||
rc=0 (exit status shown as 0 because of head pipe; the systemctl command itself fails)
|
||||
```
|
||||
With `set -e`, line 605's failure aborts the script; the linger hint (611-615) never runs.
|
||||
|
||||
Remediation verified: with ONLY `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (dir exists), `systemctl --user show-environment` succeeds. `loginctl enable-linger` is the standard persistence fix.
|
||||
|
||||
Orphaned-unit side effect confirmed: the unit file remains written even though daemon-reload failed (a real orphaned unit exists at `~/.config/systemd/user/pos-ai-server.service` from the failed attempt on this machine — see Step 6 for its corrupted ExecStart).
|
||||
|
||||
**H5 CONFIRMED.**
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Step 6: H6 — Port mismatch (downstream impact)
|
||||
|
||||
llama.cpp default port: **8080**.
|
||||
- Binary evidence: real `--help` → `--port PORT port to listen (default: 8080)`
|
||||
- Source citation: `common/common.h:620` → `int32_t port = 8080; // server listens on this network port` (also `common/common.h:261` in struct block; `from https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/common.h`)
|
||||
|
||||
Tool/adapter default: **8088** — `bin/pos-ai-server:329` (`PORT="${LLAMACPP_PORT:-8088}"`), `:281`, `:316`; `lib/ai-providers/llamacpp.sh:15,23,49` (adapter probes `http://$host:$port/v1/models` and `/v1/chat/completions`); health check `bin/pos-ai-server:198` probes `$HOST:$PORT`.
|
||||
|
||||
Real corrupted unit found on this machine (from the failed start attempt — the SIGPIPE race omitted flags):
|
||||
```
|
||||
ExecStart="/usr/local/bin/llama-server" -m "…/Qwen3-1.7B-Q8_0.gguf" --host 127.0.0.1
|
||||
```
|
||||
Only `--host` survived — `--port 8088`, `--n-gpu-layers 0`, `--ctx-size 4096`, `--threads N` were all dropped. If `systemctl` had succeeded, llama-server would bind **8080** while `check_health` and the OpenAI adapter probe **8088** → "not running" / connection refused. Even when the race lets the unit through, H6 guarantees a downstream mismatch whenever `--port` is omitted.
|
||||
|
||||
**H6 CONFIRMED.**
|
||||
|
||||
**[DONE]**
|
||||
|
||||
## Root Cause Statement (final)
|
||||
|
||||
The user's exact messages trace to four independent defects in `bin/pos-ai-server` (plus one in the adapter default):
|
||||
|
||||
1. **"installed llama.cpp unknown"** — `detect_llama_version` (`:71`) discards stderr; real llama.cpp b10822 prints `version: 0.4.0-dev (build 10822, …)` to **stderr** (`common/build-info.h:13`), so the capture is always empty → "unknown".
|
||||
2. **"does not support default flag …"** — `printf '%s' "$help_text" | grep -qE` under `set -o pipefail` races: grep -q exits at first match, printf gets SIGPIPE, pipefail promotes rc=141, so valid flags are randomly judged "unsupported" and omitted from the unit (or hard-errored when user-requested). This is the reason runs 1-2 and run 3 flagged different sets of flags — pure scheduling, not input differences.
|
||||
3. **"Model not found: Qwen-Qwen3-1.7B-GGUF"** — the HF downloader puts weights at `$HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf`, but `resolve_model` accepts only files; a directory argument fails (`:241-248`), even after the user's `mv` (the arg was still a directory name).
|
||||
4. **"Failed to connect to user scope bus …"** — the unit write (`:584-602`) precedes an unguarded `systemctl --user daemon-reload` (`:605`); under SSH neither `XDG_RUNTIME_DIR` nor `DBUS_SESSION_BUS_ADDRESS` is set, so systemctl fails, `set -e` aborts, and the linger fix hint (`:611-615`) never shows. The orphaned unit left behind can carry a corrupted ExecStart (see H6), e.g. missing `--port 8088` so the server would bind llama.cpp's default 8080 while the adapter/health check probe 8088.
|
||||
|
||||
Classification: **FACT** (mechanisms directly reproduced with the real b10822 binary; source citations for stderr stream and port default; unit artifact inspected).
|
||||
|
||||
## Fix-Point Spec
|
||||
|
||||
| # | File:line | Defect | Minimal change | Design question for Architect |
|
||||
|---|-----------|--------|----------------|-------------------------------|
|
||||
| F1 | `bin/pos-ai-server:71` | version always "unknown" (stderr discarded; regex also brittle for pure-build strings) | Capture `2>&1`; broaden regex to also accept `build [0-9]+`/`b[0-9]+`: `version="$("$bin" --version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+' | head -1 …)"` | Canonical display: semver vs build number vs both? |
|
||||
| F2 | `bin/pos-ai-server:101,145` | `printf|grep -q` + pipefail SIGPIPE race → random flag rejection | Replace the pipeline: `grep` without `-q` writing to `/dev/null`, e.g. `if grep -E -- "…" <<<"$help_text" >/dev/null; then` (non-q grep consumes all input; herestring avoids the pipe and printf EPIPE), **or** bash regex `[[ "$help_text" =~ (^|[[:space:]])${flag}([[:space:]]|=|$) ]]` | Style preference: grep-herestring vs bash `=~`; whether to harden `validate_requested_flags` identically (yes) |
|
||||
| F3 | `bin/pos-ai-server:229-259` (`resolve_model`) | directory under `HF_DOWNLOAD_DIR` (one `.gguf`) rejected | When `$candidate` is a directory: if exactly one `*.gguf` inside → use it; if multiple → list and err/ask | Should auto-expand single-gguf dirs, or require explicit file path? (Recommended: auto-expand, since downloader always produces `<repo>/<file>` layout) |
|
||||
| F4 | `bin/pos-ai-server:605-606` (+ same in matrix-listener:341, network-download:191) | no user-bus pre-check; SSH w/o XDG_RUNTIME_DIR → set -e abort; orphan unit; linger hint skipped | Pre-flight before daemon-reload: `systemctl --user show-environment` (or `printenv XDG_RUNTIME_DIR DBUS_SESSION_BUS_ADDRESS`); on failure `err` with remediation: `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if dir exists) and `sudo loginctl enable-linger $(id -un)`; optionally run the server directly (no unit) as fallback | Fallback strategy: error+hint only, or run-direct fallback? Also: consider removing the orphaned unit on failure (Builder decision) |
|
||||
| F5 | `bin/pos-ai-server:56` (`find_llamacpp`) | bare `server` / `llama.cpp/server` fallbacks can pick unrelated binaries | Drop `"server"` (keep `llama-server`, `llama-server-cuda`; keep or drop `llama.cpp/server`) | Should the fallback list be `llama-server`/`llama-server-cuda` only? |
|
||||
| F6 | `bin/pos-ai-server:528-532` + `lib/ai-providers/llamacpp.sh:15,23,49` | `--port` omission → server on 8080 vs adapter/health on 8088 | Guarantee `--port $PORT` always emitted (F2 fixes the omission); as defense-in-depth, adapter health fallback probe both 8088 and 8080, or derive from the unit | Should the tool ever allow running on llama.cpp's default 8080, or always pin 8088? |
|
||||
| F7 | `apps/ai/llamacpp.sh:46-57` | installer never verifies the installed binary runs (version/help readable) | Post-install sanity: `llama-server --version >/dev/null 2>&1 && llama-server --help >/dev/null`; warn on failure | None (simple hardening) |
|
||||
|
||||
## Design Questions (explicit)
|
||||
|
||||
1. **F2/F3 boundary:** Should unknown-version disable flag *validation* entirely (trust defaults), or retain help-based validation but fix the SIGPIPE race? (Recommended: keep validation, fix race — help-based validation is the correct design once deterministic.)
|
||||
2. **F3:** Should `resolve_model` auto-expand a single-`.gguf` directory; with multiple gguFs, err with the list?
|
||||
3. **F4:** On unresolvable user bus: error+hint only, or a run-direct fallback (no systemd unit) for headless/SSH use?
|
||||
4. **F5:** Drop the bare `server` (and possibly `llama.cpp/server`) fallback candidates?
|
||||
5. **F6:** Pin the port in the unit always (recommended) vs teach the adapter to probe the llama.cpp default 8080 as a fallback?
|
||||
|
||||
## Test-Fixture Spec (for Tester)
|
||||
|
||||
Each fix gets a stub/PATH/case that reproduces it deterministically:
|
||||
|
||||
- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit c457e3bf7)` **to stderr**, help to stdout. Assert `detect_llama_version` returns `0.4.0` (not "unknown") with the fixed `2>&1`; also fixture printing only `version: b10822`/`build 10822` to stderr to assert the broadened regex.
|
||||
- **F2 (race):** fake `llama-server` whose `--help` emits a 59 KB body with tokens `--threads` at line 7, `--ctx-size` at line 25, `--n-gpu-layers` at line 140, `--host`/`--port` near the end (exact real llama.cpp layout). Run `validate_default_flags` 20× under `set -euo pipefail`; fixed code must report all 5 supported on EVERY run (deterministic). Regression tail: pre-fix, the run must fail at least once (demonstrates the race existed).
|
||||
- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `pos ai server start Qwen-Qwen3-1.7B-GGUF` (DRY_RUN=1) resolves to the file; multi-gguf dir case errors with the file list.
|
||||
- **F4 (bus):** environment with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or a stub `systemctl` that fails with the dbus message); assert the pre-flight fails with the remediation hint AND the linger hint text when applicable; assert no orphaned unit is left (or is removed on failure).
|
||||
- **F5 (server fallback):** PATH containing ONLY a fake unrelated `server` (prints its own --help/--version) and no llama-server; assert `find_llamacpp` does NOT return `server`.
|
||||
- **F6 (port):** with F2 fixed, generate a unit with `DRY_RUN=1` and assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`).
|
||||
- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (e.g., missing shared lib) → assert post-install sanity warns.
|
||||
|
||||
## Evidence Index (concrete artifacts)
|
||||
|
||||
- Real binary downloaded/run: `llama-b10822-bin-ubuntu-x64.tar.gz` from `https://github.com/ggml-org/llama.cpp/releases/tag/b10822` (asset `llama-b10822-bin-ubuntu-x64.tar.gz`).
|
||||
- `llama-server --version` → stderr: `common/build-info.h:13` (`FILE * = stderr`), `common/arg.cpp:1454-1458`, plus byte-level `od -c`/redirect proof.
|
||||
- `--help` flag lines: real output lines 7/25/140/503/506; byte offsets 320/1892/10992/39499/39741; total 59001 bytes (< 64 KB pipe buffer).
|
||||
- Pipeline rc=141 proof: `/tmp/test_sigpipe3.sh` output (rc 0/141 alternating).
|
||||
- 10-run `pos-ai-server` flakiness table (see Step 3).
|
||||
- Real corrupted orphaned unit: `~/.config/systemd/user/pos-ai-server.service` — `ExecStart=… --host 127.0.0.1` (port/ctx/threads/gpu-layers omitted).
|
||||
- Model-not-found exact reproduction and post-`mv` reproduction.
|
||||
- User-bus error reproduced verbatim on target machine; `export XDG_RUNTIME_DIR=/run/user/$(id -u)` remediation verified working.
|
||||
- Default port: `common/common.h:620` (`int32_t port = 8080`); real `--help` `(default: 8080)`.
|
||||
- H2 fixture: fake unrelated `server` → `find_llamacpp` returns `server`, all defaults "unsupported".
|
||||
|
||||
## Next Agent
|
||||
|
||||
**Builder** — five (F1-F5) of the six fix-points are code changes in `bin/pos-ai-server` (plus `apps/ai/llamacpp.sh` for F7); Architect input needed on the design questions before/while implementing F2/F3/F4/F6.
|
||||
|
||||
**Changes made by Detective:** none (read-only; only downloaded/extracted to /tmp and created user-level fixtures under `/home/unknown/.local/share/linux_post_install/ai/models` matching the user's layout for reproduction).
|
||||
@@ -0,0 +1,75 @@
|
||||
# Maintainer — AGENT_TODO.md Done entry for 2026-09-06 AI-server round
|
||||
|
||||
## TL;DR
|
||||
|
||||
- **Finding:** AI-server breakage round (llamacpp install fallout) finished, but no
|
||||
AGENT_TODO Done entry captured it yet. Per repo convention ("move completed work
|
||||
into **Done** (dated one-line) in the same commit that finishes the work"), a
|
||||
commit is imminent and the edit must ride it. Added ONE new **Done** entry.
|
||||
- **Correction:** Added a single 2026-09-06 entry at the TOP of Done (newest-last
|
||||
placement), style matching existing entries, ~9 lines, condensing the
|
||||
Detective/Architect/Builder/Tester reports. All other content untouched.
|
||||
- **Validation:** `git diff --stat AGENT_TODO.md` = 1 file changed, 2 insertions.
|
||||
No gates run (file is not part of gen/check/lint).
|
||||
|
||||
## Step 1: Locate Done section + newest-last placement
|
||||
|
||||
Read `AGENT_TODO.md`. Done entries are newest-at-top within the section (e.g. the
|
||||
existing 2026-09-06 entries precede 2026-09-05/04/…). The new entry was inserted
|
||||
immediately after `## Done`, above the existing "Stabilization pass" entry.
|
||||
|
||||
## Step 2: Check the existing 2026-09-06 "Stabilization pass" entry for contradiction
|
||||
|
||||
Reviewed it (lines 45+). It records the 17-point audit's own scope (help-flag
|
||||
validation, security, tooling, config, tests at 12 files/179 checks). The new
|
||||
round found deeper defects (stderr version detection, SIGPIPE race, dir expansion,
|
||||
port mismatch, user-bus) that the pass did not claim to have covered. No statement
|
||||
in the stabilization entry asserts these were already fixed, so **no minimal
|
||||
adjustment is warranted** — the entries are complementary, not contradictory.
|
||||
Per the brief, left everything else untouched.
|
||||
|
||||
## Step 3: Add the new Done entry
|
||||
|
||||
Inserted one 2026-09-06 entry at the top of Done, ~9 lines, matching existing
|
||||
style (`- **YYYY-MM-DD** —` prefix, terse evidence-backed summary), condensing:
|
||||
|
||||
- Detective root cause (b10822 binary: `--version`→STDERR mg07 hidden by
|
||||
`2>/dev/null`; `printf|grep -q` SIGPIPE rc=141 flag race; `resolve_model`
|
||||
file-only vs downloader dirs; port 8080 vs 8088)
|
||||
- Architect DQ1-DQ6 (help-gated stays; no silent dir-expansion pick;
|
||||
`ensure_user_bus` pre-flight; `--no-unit` escape hatch; candidates narrowed;
|
||||
port pinned 8088; installer sanity)
|
||||
- Builder F1-F7 + F1 regex edge (`build 1.2.3`→`1`)
|
||||
- Tester (4 regression files + 3 fixture updates; 16 files / 269 checks)
|
||||
- Verification (gen idempotent, check OK, lint 0 FAIL/0 WARN, test 269/269,
|
||||
bash -n, git diff --check)
|
||||
- Post-fix user outcome (`XDG_RUNTIME_DIR` + linger → Option A or `--no-unit`)
|
||||
|
||||
[DONE]
|
||||
|
||||
## Validation
|
||||
|
||||
`git diff --stat AGENT_TODO.md`:
|
||||
|
||||
```
|
||||
AGENT_TODO.md | 2 ++
|
||||
1 file changed, 2 insertions(+)
|
||||
```
|
||||
|
||||
Only AGENT_TODO.md touched. No gates run (not part of gen/check/lint).
|
||||
|
||||
[DONE]
|
||||
|
||||
## Report status
|
||||
|
||||
Status: MAINTENANCE_COMPLETE
|
||||
|
||||
- Maintenance objective: add ONE Done entry for the AI-server round, newest-at-top,
|
||||
style-matched; adjust stale text only if it contradicts (it did not).
|
||||
- Files changed: AGENT_TODO.md (one entry added).
|
||||
- Verification: git diff stat above; content diff reviewed.
|
||||
- Scope compliance: in-scope corrections only (touched ONLY AGENT_TODO.md); no
|
||||
features, no new standards, no other files.
|
||||
- Remaining / deferred items: none for this objective. The edit rides the imminent
|
||||
commit made by the committer (I did NOT commit).
|
||||
- Recommended next agent: Orchestrator (commit + handoff).
|
||||
@@ -0,0 +1,66 @@
|
||||
# Tester report — llama-server breakage (F1–F7) regression tests
|
||||
|
||||
Date: 2026-09-06 · Tester · Objective: regression tests `tests/` for the fixed llama-server contract, `make test` evidence, no commits (per brief). Status: TESTS_READY (new files) / suite blocked by 2 pre-existing issues.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- New test files (all pass, runnable via `make test`; sandbox-only, no prod/other-test edits): `tests/t-ai-server-validate.sh` (F1/F2/F5/F6), `tests/t-ai-server-model.sh` (F3), `tests/t-ai-server-bus.sh` (F4 + E2E user/SSH scenarios), `tests/t-llamacpp-install.sh` (F7).
|
||||
- **New-suite result: 90 checks PASS / 0 FAIL / 0 SKIP, runtime ~7 s.** Total `make test` (16 files): 12 files pass / 4 files fail / 0 skip, 235 check PASS / 34 FAIL / 62 s — the 34 failures are ALL in 4 pre-existing test files, 2 root causes, NEITHER caused by my new files (verified: new files pass standalone and in the combined run).
|
||||
- **Production bug observed (REPORTED, not fixed, per brief):** F1 version regex alternation `[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+` — leftmost-match makes `grep -oE` return `build 1` (→ version `1`) for `llama.cpp build 1.2.3 (…)`. File/repro: `bin/pos-ai-server:86`; `echo "llama.cpp build 1.2.3 (abcdef)" | grep -oE '…'` → `build 1`. Breaks pre-existing `tests/t-ai-llama-detect.sh` (expects `1.2.3`, gets `1`). Real b10822 (`… (build 10822)` → `10822`) is unaffected.
|
||||
- **Fixture gap introduced by F4 pre-flight:** 3 pre-existing DRY_RUN test files (`t-ai-server-flags.sh` 26 FAIL, `t-unsupported-flags.sh` 3 FAIL, `t-config-precedence.sh` 4 FAIL) never provided a `systemctl` stub / bus env, so the new `ensure_user_bus` (called before the DRY_RUN return in `cmd_start`) aborts with rc 1 and no `ExecStart:` is emitted. NOT a production bug. Per brief I did NOT modify other tests; the fix is a fixture update (succeeding `systemctl` stub + `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` in those 3 files) → needs Orchestrator routing.
|
||||
- `make test` is therefore NOT green on the merged tree (34 fails above). After the F1 regex fix and the 3 file fixture updates, expected status: all 16 files green (my 4 + 12 already-green).
|
||||
|
||||
## Step 1: Contracts read (architect/detective/builder reports, test infra, F1–F7 code)
|
||||
|
||||
Read `AgentsReport/{architect,detective,builder}/2026-09-06_ai-server-*.md`, `tests/run-tests.sh`, `tests/test-lib.sh`, the 4 affected pre-existing tests + `t-systemd-unit.sh` (control, passes), `bin/pos-ai-server`, `lib/common.sh` `ensure_user_bus` (lines 143–154), `apps/ai/llamacpp.sh` `llamacpp_sanity` (lines 23–36) + `LLAMACPP_BIN_DIR` seam (line 17). Marked: F4 order (bus check BEFORE DRY_RUN return), `--no-unit` skips bus, `NO_UNIT_PIDFILE`/`NO_UNIT_LOG` seams, DRY_RUN unit-path output line `(dry-run) ExecStart: …`, `DRY_RUN+--no-unit` output `(dry-run) nohup …`. Pre-existing failures reproduced on the current tree before writing any new test (baseline full run: 12/4).
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 2: tests/t-ai-server-validate.sh (F1/F2/F5/F6) — 27 checks
|
||||
|
||||
Approach: `extract_fn` (brace-counting awk) extracts the REAL shipped `detect_llama_version`, `find_llamacpp`, `validate_requested_flags`, `validate_default_flags` bodies into a sandbox file sourced in `bash -c` subshells (with `$ROOT/lib/common.sh`) — no prod edit, tests the exact shipped code. 25× `validate_default_flags` loop against a real-shaped 63 864-byte / 732-line `--help` fixture (flags at lines 7/25/140/503/506, matching Detective's real llama.cpp layout, just under the 64 KB pipe buffer — the adversarial size that made the pre-fix `printf|grep -q` race observable). Each run asserts the whole invocation rc=0 AND all five globals `1|1|1|1|1` (catches an rc=141 SIGPIPE component explicitly). 20× requested-flag cases (`--port` accepted; `--tensor-split` rejected naming `0.4.0`); F1 stderr-vs-file cases (`stderr` → `0.4.0`, `build 10822` → `10822`, `b10822` → `b10822`, file-read ok, missing binary → `unknown` rc 0; all real CLI). F5: bare `server` no longer a candidate (only `llama-server`/`llama-server-cuda`), `llama-server` preferred. F6: DRY_RUN `start` (succeeding `systemctl` stub, real CLI) → ExecStart has `--port 8088 --host 127.0.0.1 --n-gpu-layers 0 --ctx-size 4096 --threads`; `status` shows `version: 0.4.0`, never `unknown`.
|
||||
|
||||
Result: 27 PASS / 0 FAIL / 0 SKIP, ~3 s.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 3: tests/t-ai-server-model.sh (F3) — 18 checks
|
||||
|
||||
Unit tests of real `resolve_model`/`resolve_gguf_in_dir` (HF layout fixture: `models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf`, multi dir, empty dir, flat file, absolute path) via `bash -c "source '$com'; source '$fn_file'; resolve_model '<arg>'";` with `HF_DOWNLOAD_DIR` env; plus CLI integration through real `pos-ai-server` (`start … --no-unit` DRY_RUN resolves to `-m "…/Qwen3-1.7B-Q8_0.gguf"`; multi-dir CLI start errs `pick one` rc 1). First run had 4 failures in the multi/empty unit cases (`err`/`pick_model` not found in the un-sourced subshell); fixed by sourcing `$com` in every subshell — re-run 18/18.
|
||||
|
||||
Result: 18 PASS / 0 FAIL / 0 SKIP, ~1 s.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 4: tests/t-ai-server-bus.sh (F4 + E2E) — 36 checks
|
||||
|
||||
Stub farm (install-shaped `llama-server` with b10822 stderr version + full-support help; failing `systemctl` = bus-missing SSH shape; `nvidia-smi` exit 1 → deterministic CPU path; `curl` → `{"status":"ok"}`; `loginctl` → `Linger=no`; env `-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS` for bus-less cases since the outer shell has them unset). Verified: bus missing + real `start` → rc≠0, output has BOTH remediation lines + `no unit was written`, NO orphaned unit file; bus present (env + succeeding stub) + DRY_RUN → rc 0 and proceeds to `ExecStart:`; `--no-unit` bus-less escape hatch → real start writes pidfile, launches stub (`exec sleep 300` so the logged pid IS the sleeper — no orphans; stop killed it, no stray processes after suite), `status` finds `service: running` via pidfile, `stop` kills + removes pidfile; E2E user scenario (bus up, unit path): `start Qwen-Qwen3-1.7B-GGUF` rc 0, unit written, ExecStart has model + `--port 8088 --host 127.0.0.1 --n-gpu-layers 0 --ctx-size 4096 --threads`, `status` reports `version: 0.4.0` not `unknown`, `systemd-analyze verify` accepts the unit (BEFORE-LINGER/WARN tolerances confirmed on systemd 257), `stop` removes unit; E2E SSH-shaped: rc 1, both remediation lines, no unit. The `printf '...%s...'` with `--`-leading strings lesson from Step 2 also applies to the unit ExecStart assertions — used `printf '%s\n'` consistently.
|
||||
|
||||
Result: 36 PASS / 0 FAIL / 0 SKIP, ~4 s.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 5: tests/t-llamacpp-install.sh (F7) — 9 checks
|
||||
|
||||
Extracts real shipped `llamacpp_sanity` via `extract_fn`; exercises through the `LLAMACPP_BIN_DIR` seam with the bin dir on PATH. Healthy fixture → rc 0 + `llama.cpp sanity OK`; dangling symlink → rc 1 + `dangling symlink`; non-executable file → rc 1 with a sanity err (bash `command -v` finds the file by PATH existence, so the failure surfaces at the `--version` exec step: `did not run`); broken binary (`--version` exit 1, missing-shared-lib shape) → rc 1 `did not run`; static guard: shipped installer wires `llamacpp_sanity` into install. Fixed one assertion after the first run: my expected `not on PATH` message was wrong for the non-executable case (real behavior: `command -v` exists-check passes, exec fails) — re-run 9/9.
|
||||
|
||||
Result: 9 PASS / 0 FAIL / 0 SKIP, <1 s.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 6: Full-suite evidence
|
||||
|
||||
- My 4 files together: `./tests/run-tests.sh t-ai-server-validate.sh t-ai-server-model.sh t-ai-server-bus.sh t-llamacpp-install.sh` → files 4 pass / 0 fail / 0 skip; checks 90 pass / 0 fail / 0 skip; runtime 7 s (fits the ≤30 s new-suite budget).
|
||||
- `make test` (16 files): files 12 pass / 4 fail / 0 skip; checks 235 pass / 34 fail / 0 skip; runtime 62 s (< 90 s budget).
|
||||
- Fail breakdown (all pre-existing files): `t-ai-llama-detect.sh` 8 checks, 1 FAIL — F1 regex production bug (`version: 1` vs expected `1.2.3`); `t-ai-server-flags.sh` 28 checks, 26 FAIL — bus-gap (all 26 abort rc 1 before `ExecStart:`); `t-config-precedence.sh` 39 checks, 4 FAIL — bus-gap (B1–B4, empty ExecStart); `t-unsupported-flags.sh` 16 checks, 3 FAIL — bus-gap (rc 1 + missing `--threads` warn because start aborted before validation path).
|
||||
- `bash -n` clean on all 4 new files. No stray processes/pidfiles after the `--no-unit` case (`pgrep -af 'llama-server|sleep 300'` empty, no `/tmp/pos-ai-server.pid`).
|
||||
- git status: 4 new `tests/t-*.sh` untracked (not committed per brief); working tree still carries Builder's uncommitted F1–F7 changes.
|
||||
|
||||
[PASS] (new files) · suite-level blockers in TL;DR
|
||||
|
||||
## Handoff / blockers
|
||||
|
||||
- **F1 production bug** → Builder: fix `bin/pos-ai-server:86` alternation ordering (e.g. put `[0-9]+\.[0-9]+\.[0-9]+` first or drop `build [0-9]+`, matching real b10822 semantics); then `t-ai-llama-detect.sh` should go green (verify).
|
||||
- **F4 bus-gap fixtures** → Orchestrator to authorize editing the 3 pre-existing DRY_RUN test files (`t-ai-server-flags.sh`, `t-unsupported-flags.sh`, `t-config-precedence.sh`): add a succeeding `systemctl` stub + set `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` in their `start` invocations (pattern: `tests/t-systemd-unit.sh` + my `t-ai-server-bus.sh`). Once authorized, this is a small Maintainer/Tester fixture edit, not a prod change.
|
||||
- After both: expect all 16 files green — 269 checks = my 90 + 145 pre-existing passes + the 34 fixed. Reviewer can then independently re-run `make test`.
|
||||
- Residual risk (documented, not blocking): F1 regex fragility for `build X.Y.Z-dev` strings remains until the production fix; my F1 tests assert real b10822-shaped output which the current code handles.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Busgap Fixture Update — 2026-09-06
|
||||
|
||||
## TL;DR
|
||||
|
||||
- **F4 (`ensure_user_bus`)** in `bin/pos-ai-server` now runs in `cmd_start` BEFORE the DRY_RUN return (verified in `lib/common.sh:148` + `bin/pos-ai-server:654-671`), so every unit-path `start` without a reachable user systemd bus aborts with rc 1 and no `ExecStart:` output.
|
||||
- Three pre-existing DRY_RUN test files never provided a `systemctl` stub / bus env, so 33 checks failed (26 + 3 + 4).
|
||||
- **Fix:** added a succeeding `systemctl` stub (return 0) to each affected sandbox, matching the established pattern in `t-systemd-unit.sh` and `t-ai-server-bus.sh`. No assertions weakened; bus-missing behavior remains covered in `t-ai-server-bus.sh`.
|
||||
- **Result:** full suite green — **16 files pass / 0 fail, 269 checks pass / 0 fail, 0 skip. Runtime 72s.**
|
||||
|
||||
**PASS/FAIL totals: 269 pass / 0 fail. Defects by severity: none.**
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Establish failure & mechanism
|
||||
|
||||
Read `bin/pos-ai-server`, `lib/common.sh`, `tests/run-tests.sh`, the 3 affected tests, and the passing control tests (`t-systemd-unit.sh`, `t-ai-server-bus.sh`).
|
||||
|
||||
**Verified call order:** `cmd_start` (pos-ai-server:654-655) calls `ensure_user_bus` when `NO_UNIT != 1`, i.e. **before** the `DRY_RUN` early-return at line 661. `ensure_user_bus` (common.sh:148-154) runs `systemctl --user show-environment &>/dev/null` and `err`s on nonzero. All three DRY_RUN tests drive the unit path (no `--no-unit`), so without a `systemctl` stub they abort with rc 1 and no `ExecStart:` output.
|
||||
|
||||
## Step 2: Apply fixtures (systemctl stub returning 0)
|
||||
|
||||
Established pattern: `printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"` + add to `chmod +x` (exact copy of `t-systemd-unit.sh:41,46`). Also tested that this satisfies the `systemctl --user show-environment` probe without needing `XDG_RUNTIME_DIR` — the stub returns 0, so the pre-flight passes. Assertion lines untouched.
|
||||
|
||||
- `t-ai-server-flags.sh`: added stub after nvidia-smi, extended chmod.
|
||||
- `t-unsupported-flags.sh`: added stub after first llama-server/chmod, extended chmod.
|
||||
- `t-config-precedence.sh` (Part B): added stub after nvidia-smi, extended chmod.
|
||||
|
||||
[PASS]
|
||||
|
||||
## Step 3: bash -n + full suite
|
||||
|
||||
- `bash -n` on all three edited files: OK.
|
||||
- `make test`: 16 files / 269 checks, all pass.
|
||||
|
||||
Counts per modified file: `t-ai-server-flags.sh` 28, `t-config-precedence.sh` 43, `t-unsupported-flags.sh` 19. (Prior failing counts: 26 / 4 / 3. The higher post-fix counts reflect the checks now actually running to completion instead of aborting on the bus error; the pre-existing assertion set was preserved.)
|
||||
|
||||
[PASS]
|
||||
|
||||
---
|
||||
|
||||
## Handoff
|
||||
|
||||
**Status:** TESTS_READY
|
||||
|
||||
- **Files changed (test fixtures only):** `tests/t-ai-server-flags.sh`, `tests/t-unsupported-flags.sh`, `tests/t-config-precedence.sh`
|
||||
- **Assertions:** unchanged (only added a systemctl stub + chmod); bus-missing behavior stays in `t-ai-server-bus.sh`.
|
||||
- **Verification:** `bash -n` all three; `make test` → 269 pass / 0 fail, 72s.
|
||||
- **No production code changed; nothing committed.**
|
||||
- **Residual failures:** none. (Noted: F1-regex fix and these fixtures land in parallel; this run was green, so no rerun needed.)
|
||||
@@ -371,7 +371,7 @@ source "$(dirname "$0")/../lib/common.sh"
|
||||
|
||||
**Scripts that do NOT source common.sh** (self-contained):
|
||||
<!-- GEN:START selfcontained -->
|
||||
`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-listener`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`.
|
||||
`pos`, `pos-ai-gemini`, `pos-ai-llamacpp`, `pos-ai-openrouter`, `pos-communication-matrix-sender`, `pos-communication-telegram-listener`, `pos-communication-telegram-sender`, `pos-network-checkport`, `pos-network-hotspot`, `pos-network-ip`, `pos-network-scan`, `pos-ssh-load-keys`, `pos-system-firewall`.
|
||||
<!-- GEN:END selfcontained -->
|
||||
|
||||
---
|
||||
@@ -618,8 +618,8 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-ai-hf` | 993 | Download AI models from Hugging Face (search, download, manage) |
|
||||
| `bin/pos-ai-llamacpp` | 7 | Forward to pos ai --provider llamacpp (backward compat) |
|
||||
| `bin/pos-ai-openrouter` | 7 | Forward to pos ai --provider openrouter (backward compat) |
|
||||
| `bin/pos-ai-server` | 743 | llama.cpp local inference server (start, stop, status, models, logs) |
|
||||
| `bin/pos-communication-matrix-listener` | 567 | Matrix listener: map /command → bash, run them on room messages |
|
||||
| `bin/pos-ai-server` | 879 | llama.cpp local inference server (start, stop, status, models, logs) |
|
||||
| `bin/pos-communication-matrix-listener` | 582 | Matrix listener: map /command → bash, run them on room messages |
|
||||
| `bin/pos-communication-matrix-sender` | 215 | Send messages to a Matrix room via the client-server API (send, test, login) |
|
||||
| `bin/pos-communication-scrcpy` | 245 | Mirror/control an Android device via scrcpy+adb (mirror, devices, record, tcpip, connect, push, pull, screenshot, info) |
|
||||
| `bin/pos-communication-telegram-listener` | 815 | Telegram bot listener: map /command → bash and <prefix> → app, run them on chat messages |
|
||||
@@ -640,7 +640,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-media-sync` | 219 | Incremental Music → USB sync (mp3/mp4, add/update only) |
|
||||
| `bin/pos-media-ytsync` | 1213 | Incrementally sync YouTube channels/playlists into ~/Videos |
|
||||
| `bin/pos-network-checkport` | 498 | Check TCP/UDP port reachability (nmap, or bash/nc fallback) + local interface view |
|
||||
| `bin/pos-network-download` | 1110 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) |
|
||||
| `bin/pos-network-download` | 1113 | aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) |
|
||||
| `bin/pos-network-hotspot` | 93 | Wi-Fi hotspot via create_ap + wihotspot-gui |
|
||||
| `bin/pos-network-ip` | 69 | Show interfaces, routes, public IP + location |
|
||||
| `bin/pos-network-scan` | 272 | Parallel ping sweep of CIDR |
|
||||
|
||||
+1
-1
@@ -124,7 +124,7 @@ Auth: `HF_TOKEN` in `~/.config/linux_post_install/ai.env` (same scope as `pos ai
|
||||
| `pos ai server models` | List `.gguf` files found in `HF_DOWNLOAD_DIR` with sizes |
|
||||
| `pos ai server logs [lines]` | Show recent server logs via `journalctl --user -u pos-ai-server` (default 50 lines) |
|
||||
|
||||
Flags: `--port <port>` (default 8088), `--host <addr>` (default 127.0.0.1), `--model <path>` (overrides arg/config), `--ctx <size>` (context window, default 4096), `--gpu <layers>` (-1=auto, 0=CPU, N=explicit, default -1), `--threads <n>` (default nproc), `--gpu-layers`/`--n-gpu-layers <n>` (GPU layers override), `--gpu-threads <n>`, `--tensor-split <n>`, `--batch-size <n>`, `--ubatch-size <n>`, `--temperature <n>`, `--top-k <n>`, `--top-p <n>`, `--repetition-penalty <n>`, `--mmap`, `--mlock`, `--kv-cache <size>`, `--ctx-size <n>`, `--metrics`, `--health`, `--slots <n>`. **Every flag that enters `ExecStart` is validated** against the installed llama.cpp's `--help` (word-boundary match, version-aware). Flag validation is split by intent: flags the user explicitly **requested** (via CLI, `LLAMACPP_*` config, or exported env) that are unsupported cause a **hard error** naming the flag + detected version; always-emitted **default** flags (`--port`, `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`) that the user did not request and that are unsupported are **omitted from the unit with a single warning** (never a hard error, never silently passing an unsupported flag). If `--help` cannot be read the tool warns and proceeds (all flags accepted). The generated unit and `--dry-run` contain only flags that passed validation. Config keys in `ai.env`: `LLAMACPP_PORT`, `LLAMACPP_HOST`, `LLAMACPP_MODEL`, `LLAMACPP_CTX_SIZE`, `LLAMACPP_GPU_LAYERS`, `LLAMACPP_THREADS`. Requires `curl` + `jq` and a `llama-server` binary on PATH (install via `bash apps/install.sh llamacpp`).
|
||||
Flags: `--port <port>` (default 8088), `--host <addr>` (default 127.0.0.1), `--model <path>` (overrides arg/config), `--ctx <size>` (context window, default 4096), `--gpu <layers>` (-1=auto, 0=CPU, N=explicit, default -1), `--threads <n>` (default nproc), `--gpu-layers`/`--n-gpu-layers <n>` (GPU layers override), `--gpu-threads <n>`, `--tensor-split <n>`, `--batch-size <n>`, `--ubatch-size <n>`, `--temperature <n>`, `--top-k <n>`, `--top-p <n>`, `--repetition-penalty <n>`, `--mmap`, `--mlock`, `--kv-cache <size>`, `--ctx-size <n>`, `--metrics`, `--health`, `--slots <n>`, `--no-unit` (run the server directly under nohup with a pidfile in `$XDG_RUNTIME_DIR`/`/tmp` instead of installing a systemd unit — for headless/SSH boxes whose user systemd bus is unreachable; `stop`/`status` still work via the pidfile). Without `--no-unit`, `start` pre-flights the user systemd bus (`ensure_user_bus`) and aborts with remediation (`export XDG_RUNTIME_DIR=…`, `sudo loginctl enable-linger …`) before writing any unit. **Every flag that enters `ExecStart` is validated** against the installed llama.cpp's `--help` (word-boundary match, version-aware). Flag validation is split by intent: flags the user explicitly **requested** (via CLI, `LLAMACPP_*` config, or exported env) that are unsupported cause a **hard error** naming the flag + detected version; always-emitted **default** flags (`--port`, `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`) that the user did not request and that are unsupported are **omitted from the unit with a single warning** (never a hard error, never silently passing an unsupported flag). If `--help` cannot be read the tool warns and proceeds (all flags accepted). The generated unit and `--dry-run` contain only flags that passed validation. Config keys in `ai.env`: `LLAMACPP_PORT`, `LLAMACPP_HOST`, `LLAMACPP_MODEL`, `LLAMACPP_CTX_SIZE`, `LLAMACPP_GPU_LAYERS`, `LLAMACPP_THREADS`. Requires `curl` + `jq` and a `llama-server` binary on PATH (install via `bash apps/install.sh llamacpp`).
|
||||
|
||||
### network
|
||||
|
||||
|
||||
@@ -12,6 +12,29 @@ source "$(dirname "$0")/../../lib/common.sh"
|
||||
# instead of hitting /releases/latest.
|
||||
RELEASES_URL="https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=10"
|
||||
|
||||
# Test seam for the post-install sanity (defaults to the real install target);
|
||||
# PATH must still contain the dir for the `command -v` check.
|
||||
LLAMACPP_BIN_DIR="${LLAMACPP_BIN_DIR:-/usr/local/bin}"
|
||||
|
||||
# Post-install sanity (F7): fail fast on a genuinely broken install — missing
|
||||
# shared lib (binary won't execute) or a truncated archive (dangling symlink) —
|
||||
# with one clear err, instead of "version unknown / flags rejected" on the
|
||||
# first `pos ai server start`.
|
||||
llamacpp_sanity() {
|
||||
local bin="$LLAMACPP_BIN_DIR/llama-server"
|
||||
[ -e "$bin" ] \
|
||||
|| err "llama.cpp install sanity failed: $bin is missing or a dangling symlink (truncated archive?)"
|
||||
command -v llama-server >/dev/null 2>&1 \
|
||||
|| err "llama.cpp install sanity failed: llama-server not on PATH — check that $LLAMACPP_BIN_DIR is in PATH"
|
||||
# llama.cpp prints --version to STDERR (common/build-info.h), so 2>&1 is
|
||||
# required to actually exercise the stream the tool chain reads.
|
||||
llama-server --version >/dev/null 2>&1 \
|
||||
|| err "llama.cpp install sanity failed: 'llama-server --version' did not run — missing shared library or truncated archive"
|
||||
llama-server --help >/dev/null 2>&1 \
|
||||
|| err "llama.cpp install sanity failed: 'llama-server --help' did not run — missing shared library or truncated archive"
|
||||
log "llama.cpp sanity OK — llama-server runs (version/help readable, symlink target present)"
|
||||
}
|
||||
|
||||
install_llamacpp() {
|
||||
command -v llama-server &>/dev/null && { log "llama.cpp already installed"; return 0; }
|
||||
|
||||
@@ -56,6 +79,8 @@ sys.exit(1)
|
||||
rm -f /tmp/llamacpp.tar.gz /tmp/llamacpp-releases.json
|
||||
"
|
||||
|
||||
llamacpp_sanity
|
||||
|
||||
log "llama.cpp $tag installed — run the server with 'pos ai server start <model.gguf>'"
|
||||
}
|
||||
|
||||
|
||||
+152
-16
@@ -2,7 +2,7 @@
|
||||
set -euo pipefail
|
||||
# POS: ai server — llama.cpp local inference server (start, stop, status, models, logs)
|
||||
# POS_SUBCMDS: start stop status models logs
|
||||
# POS_FLAGS: --port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots
|
||||
# POS_FLAGS: --port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots --no-unit
|
||||
# POS_DEPS: curl jq
|
||||
|
||||
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
|
||||
@@ -20,6 +20,12 @@ USER_SYSTEMD_DIR="${USER_SYSTEMD_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/systemd/
|
||||
SERVICE="pos-ai-server.service"
|
||||
HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}"
|
||||
|
||||
# Direct-run (--no-unit) escape hatch for headless/SSH boxes without a user
|
||||
# systemd bus: pidfile + log live under $XDG_RUNTIME_DIR when set, else /tmp.
|
||||
NO_UNIT=0
|
||||
NO_UNIT_LOG="${NO_UNIT_LOG:-${XDG_RUNTIME_DIR:-/tmp}/pos-ai-server.log}"
|
||||
NO_UNIT_PIDFILE="${NO_UNIT_PIDFILE:-${XDG_RUNTIME_DIR:-/tmp}/pos-ai-server.pid}"
|
||||
|
||||
# ── Config loader (canonical env-var precedence, same pattern as pos-ai-hf) ──
|
||||
# Loaded keys are also recorded in LOADED_ENV_KEYS (see below); the D-F
|
||||
# requested-from-config tracking uses the exported values, so it is
|
||||
@@ -53,7 +59,11 @@ requested_from_env_config LLAMACPP_THREADS --threads
|
||||
|
||||
# ── Binary detection ───────────────────────────────────────────
|
||||
find_llamacpp() {
|
||||
local candidates=("llama-server" "llama.cpp/server" "server" "llama-server-cuda")
|
||||
# Deliberately narrow (F5): the bare `server` candidate previously picked
|
||||
# an UNRELATED on-PATH binary named `server`, after which every default
|
||||
# flag was rejected as "unsupported". Anonymous `llama.cpp/server` is not
|
||||
# a real on-PATH name either. Only real llama.cpp server names remain.
|
||||
local candidates=("llama-server" "llama-server-cuda")
|
||||
local bin
|
||||
for bin in "${candidates[@]}"; do
|
||||
command -v "$bin" &>/dev/null && { echo "$bin"; return 0; }
|
||||
@@ -62,13 +72,19 @@ find_llamacpp() {
|
||||
}
|
||||
|
||||
# ── Version detection ──────────────────────────────────────────
|
||||
# detect_llama_version <binary> → X.Y.Z or "unknown". Guarded: a missing
|
||||
# binary or unreadable --version output yields "unknown", never an errexit.
|
||||
# detect_llama_version <binary> → X.Y.Z, a build token, or "unknown". Guarded:
|
||||
# a missing binary or unreadable --version output yields "unknown", never an
|
||||
# errexit. Real llama.cpp prints `version: 0.4.0-dev (build 10822, commit …)`
|
||||
# to STDERR (common/build-info.h), so both streams are captured; the regex
|
||||
# accepts semver OR a build token (`build 10822` / `b10822`) so very old
|
||||
# builds that print no semver still resolve. The build number is displayed
|
||||
# without the "build " phrase.
|
||||
detect_llama_version() {
|
||||
local bin="${1:-llama-server}"
|
||||
command -v "$bin" &>/dev/null || { echo "unknown"; return 0; }
|
||||
local version
|
||||
version="$("$bin" --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
|
||||
version="$("$bin" --version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+(\.[0-9]+)*|b[0-9]+' | head -1 || true)"
|
||||
version="${version#build }"
|
||||
[ -n "$version" ] || version="unknown"
|
||||
echo "$version"
|
||||
}
|
||||
@@ -97,8 +113,12 @@ validate_requested_flags() {
|
||||
seen+=("$flag")
|
||||
# Word-boundary match: the flag token must appear as a whole word in
|
||||
# --help, not as a substring of a longer flag (D4 — e.g. --mmap must
|
||||
# not match a --no-mmap entry).
|
||||
if ! printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then
|
||||
# not match a --no-mmap entry). NON-quiet grep: with `grep -q` under
|
||||
# set -o pipefail, grep exits at the first match and printf dies of
|
||||
# SIGPIPE → rc=141 → valid flags randomly judged unsupported (F2).
|
||||
# Here ``<<<`` needs no pipe and non-q grep consumes the whole input,
|
||||
# so no early-exit race exists.
|
||||
if ! grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null; then
|
||||
err "installed llama.cpp ${version} does not expose ${flag} — remove it or upgrade llama.cpp"
|
||||
fi
|
||||
done
|
||||
@@ -142,7 +162,9 @@ validate_default_flags() {
|
||||
case "$requested_str" in
|
||||
*" $flag "*) continue ;; # requested → already hard-validated, keep
|
||||
esac
|
||||
if printf '%s' "$help_text" | grep -qE -- "(^|[[:space:]])${flag}([[:space:]]|=|$)"; then
|
||||
# NON-quiet grep (see validate_requested_flags — F2): no pipe, no
|
||||
# early-grep-exit, no printf SIGPIPE/rc=141 race under pipefail.
|
||||
if grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null; then
|
||||
eval "$ok_var=1"
|
||||
else
|
||||
warn "installed llama.cpp ${version} does not support default flag ${flag} — omitting it from the unit"
|
||||
@@ -227,15 +249,49 @@ pick_model() {
|
||||
}
|
||||
|
||||
# ── Model resolution ───────────────────────────────────────────
|
||||
# resolve_gguf_in_dir <dir> — resolve a model DIRECTORY to its single top-level
|
||||
# *.gguf (case-insensitive, no recursion, matching the HF downloader's
|
||||
# $HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf layout). Exactly one → print it and
|
||||
# return 0. Multiple → list every candidate as <dirname>/<file> on stderr and
|
||||
# err "pick one" — NEVER silently pick. Zero → return 1 (caller falls through
|
||||
# to the standard "Model not found" error).
|
||||
resolve_gguf_in_dir() {
|
||||
local dir="$1" matches=() f
|
||||
while IFS= read -r f; do
|
||||
[ -f "$f" ] && matches+=("$f")
|
||||
done < <(find "$dir" -maxdepth 1 -type f -iname '*.gguf' 2>/dev/null | sort)
|
||||
if [ "${#matches[@]}" -eq 1 ]; then
|
||||
printf '%s' "${matches[0]}"
|
||||
return 0
|
||||
fi
|
||||
if [ "${#matches[@]}" -gt 1 ]; then
|
||||
local dirname="${dir##*/}"
|
||||
for f in "${matches[@]}"; do
|
||||
echo " $dirname/$(basename "$f")" >&2
|
||||
done
|
||||
err "model dir $dirname contains multiple .gguf files — pick one (e.g. 'pos ai server start $dirname/<file>.gguf')"
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
resolve_model() {
|
||||
local explicit="${1:-}"
|
||||
# 1. Explicit argument
|
||||
if [ -n "$explicit" ]; then
|
||||
# Absolute path
|
||||
if [[ "$explicit" == /* ]]; then
|
||||
[ -f "$explicit" ] || err "Model not found: $explicit"
|
||||
printf '%s' "$explicit"
|
||||
return
|
||||
[ -e "$explicit" ] || err "Model not found: $explicit"
|
||||
if [ -f "$explicit" ]; then
|
||||
printf '%s' "$explicit"
|
||||
return
|
||||
fi
|
||||
# Absolute DIRECTORY — same single-.gguf expansion as below.
|
||||
local abs_resolved
|
||||
if abs_resolved="$(resolve_gguf_in_dir "$explicit")"; then
|
||||
printf '%s' "$abs_resolved"
|
||||
return
|
||||
fi
|
||||
err "Model not found: $explicit"
|
||||
fi
|
||||
# Relative to HF_DOWNLOAD_DIR
|
||||
local candidate="$HF_DOWNLOAD_DIR/$explicit"
|
||||
@@ -243,6 +299,17 @@ resolve_model() {
|
||||
printf '%s' "$candidate"
|
||||
return
|
||||
fi
|
||||
# Directory under HF_DOWNLOAD_DIR (repo slug form) — the HF downloader
|
||||
# writes $HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf; a single top-level
|
||||
# .gguf resolves to it, multiple err with the pick-one list, zero
|
||||
# falls through to the not-found error below (F3).
|
||||
if [ -d "$candidate" ]; then
|
||||
local slug_resolved
|
||||
if slug_resolved="$(resolve_gguf_in_dir "$candidate")"; then
|
||||
printf '%s' "$slug_resolved"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
# Also try with the name as-is (could be a relative path)
|
||||
[ -f "$explicit" ] && { printf '%s' "$explicit"; return; }
|
||||
err "Model not found: $explicit (also searched $HF_DOWNLOAD_DIR)"
|
||||
@@ -301,6 +368,9 @@ Options:
|
||||
--metrics Enable metrics endpoint
|
||||
--health Enable health endpoint
|
||||
--slots <n> Concurrent request slots
|
||||
--no-unit Run the server directly (nohup + pidfile) instead of
|
||||
installing a systemd unit — for headless/SSH boxes whose
|
||||
user systemd bus is unreachable; stop/status still work
|
||||
|
||||
Examples:
|
||||
pos ai server start mistral-7b-v0.1.Q4_K_M.gguf
|
||||
@@ -424,6 +494,8 @@ while [ $# -gt 0 ]; do
|
||||
--slots)
|
||||
[ $# -ge 2 ] || err "--slots requires a value"
|
||||
SLOTS="$2"; REQUESTED_FLAGS+=("--slots"); shift 2 ;;
|
||||
--no-unit)
|
||||
NO_UNIT=1; shift ;;
|
||||
-*)
|
||||
err "Unknown option '$1' (see --help)" ;;
|
||||
*)
|
||||
@@ -573,10 +645,47 @@ cmd_start() {
|
||||
exec_cmd+=" --slots $SLOTS"
|
||||
fi
|
||||
|
||||
# ── User-bus pre-flight + stale-unit warning (unit path only) ──
|
||||
# ensure_user_bus aborts with remediation BEFORE the unit is written, so a
|
||||
# broken user bus leaves NO orphaned unit. --no-unit skips the bus check —
|
||||
# that is its whole purpose (headless/SSH boxes without a user bus). An
|
||||
# existing unit is warned about but never deleted here (removal stays in
|
||||
# cmd_stop); it may be stale from a previous failed start.
|
||||
if [ "$NO_UNIT" -ne 1 ]; then
|
||||
ensure_user_bus
|
||||
if [ -f "$USER_SYSTEMD_DIR/$SERVICE" ]; then
|
||||
warn "existing unit $USER_SYSTEMD_DIR/$SERVICE will be overwritten — it may be stale from a previous failed start"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "${DRY_RUN:-0}" -eq 1 ]; then
|
||||
log "(dry-run) generate systemd unit $USER_SYSTEMD_DIR/$SERVICE"
|
||||
log "(dry-run) ExecStart: $exec_cmd"
|
||||
log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE"
|
||||
if [ "$NO_UNIT" -eq 1 ]; then
|
||||
log "(dry-run) nohup $exec_cmd >$NO_UNIT_LOG 2>&1 &"
|
||||
log "(dry-run) pidfile: $NO_UNIT_PIDFILE"
|
||||
else
|
||||
log "(dry-run) generate systemd unit $USER_SYSTEMD_DIR/$SERVICE"
|
||||
log "(dry-run) ExecStart: $exec_cmd"
|
||||
log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
# ── Direct run (--no-unit) ──
|
||||
# Run the exact ExecStart command directly under nohup with a pidfile
|
||||
# (under $XDG_RUNTIME_DIR, else /tmp) so stop/status keep working without
|
||||
# a systemd unit. The string is the same one the unit would carry; only
|
||||
# the binary and model tokens are quoted, and both are validated paths
|
||||
# (command -v / existing file) resolved above, so eval is the faithful
|
||||
# way to honor its systemd-style quoting.
|
||||
if [ "$NO_UNIT" -eq 1 ]; then
|
||||
mkdir -p "$(dirname "$NO_UNIT_PIDFILE")"
|
||||
: >"$NO_UNIT_LOG"
|
||||
eval "nohup $exec_cmd >'$NO_UNIT_LOG' 2>&1 &"
|
||||
local direct_pid=$!
|
||||
printf '%s\n' "$direct_pid" >"$NO_UNIT_PIDFILE"
|
||||
log "Server starting (direct run, no systemd) — pid $direct_pid, model: $(basename "$model"), port: $PORT"
|
||||
log "log: $NO_UNIT_LOG"
|
||||
log "stop: kill \$(cat $NO_UNIT_PIDFILE) (or 'pos ai server stop')"
|
||||
return 0
|
||||
fi
|
||||
|
||||
@@ -626,8 +735,28 @@ EOF
|
||||
}
|
||||
|
||||
cmd_stop() {
|
||||
# Direct-run (--no-unit) pidfile first: kill the recorded pid, remove the
|
||||
# pidfile. Then fall through to the normal unit path (both can exist if a
|
||||
# unit was installed after a direct run).
|
||||
local stopped_direct=0
|
||||
if [ -f "$NO_UNIT_PIDFILE" ]; then
|
||||
stopped_direct=1
|
||||
local direct_pid
|
||||
direct_pid="$(cat "$NO_UNIT_PIDFILE" 2>/dev/null || true)"
|
||||
if [ "${DRY_RUN:-0}" -eq 1 ]; then
|
||||
log "(dry-run) kill $direct_pid; rm -f $NO_UNIT_PIDFILE"
|
||||
else
|
||||
if [[ "$direct_pid" =~ ^[0-9]+$ ]]; then
|
||||
kill "$direct_pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$NO_UNIT_PIDFILE"
|
||||
log "llama.cpp server stopped (direct run)"
|
||||
fi
|
||||
fi
|
||||
if [ ! -f "$USER_SYSTEMD_DIR/$SERVICE" ]; then
|
||||
warn "No llama.cpp server service installed ($SERVICE)"
|
||||
if [ "$stopped_direct" -eq 0 ]; then
|
||||
warn "No llama.cpp server service installed ($SERVICE)"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
if [ "${DRY_RUN:-0}" -eq 1 ]; then
|
||||
@@ -647,8 +776,15 @@ cmd_status() {
|
||||
err "llama-server not found — install llama.cpp with the app installer: 'apps/ai/llamacpp.sh' (run 'bash apps/install.sh llamacpp', or pass '--apps'/'--full' to install.sh), see 'pos help ai server' (https://github.com/ggerganov/llama.cpp)"
|
||||
fi
|
||||
|
||||
# Service state
|
||||
# Service state — systemd unit, or the direct-run (--no-unit) pidfile
|
||||
local svc_state="stopped"
|
||||
if [ -f "$NO_UNIT_PIDFILE" ]; then
|
||||
local direct_pid
|
||||
direct_pid="$(cat "$NO_UNIT_PIDFILE" 2>/dev/null || true)"
|
||||
if [[ "$direct_pid" =~ ^[0-9]+$ ]] && kill -0 "$direct_pid" 2>/dev/null; then
|
||||
svc_state="running"
|
||||
fi
|
||||
fi
|
||||
if systemctl --user is-active "$SERVICE" &>/dev/null; then
|
||||
svc_state="running"
|
||||
fi
|
||||
|
||||
@@ -20,6 +20,18 @@ err() { echo "ERROR: $*" >&2; exit 1; }
|
||||
log() { echo "[+] $*"; }
|
||||
warn() { echo "[!] $*" >&2; }
|
||||
|
||||
# User-bus pre-flight — identical remediation to lib/common.sh's
|
||||
# ensure_user_bus; this tool uses inline fallbacks instead of sourcing
|
||||
# common.sh, so the helper is duplicated (guarded in case common.sh is ever
|
||||
# sourced too).
|
||||
declare -F ensure_user_bus >/dev/null || ensure_user_bus() {
|
||||
systemctl --user show-environment &>/dev/null && return 0
|
||||
err "cannot reach the user systemd bus (common in SSH/headless sessions) — systemctl --user failed
|
||||
fix: export XDG_RUNTIME_DIR=/run/user/$(id -u) (if the directory exists)
|
||||
fix: sudo loginctl enable-linger $(id -un) (persist the user session so the bus survives logouts)
|
||||
no unit was written — fix the bus and retry"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: pos communication matrix listener [command]
|
||||
@@ -306,6 +318,9 @@ ui() {
|
||||
# ── systemd user service ────────────────────────────────────────
|
||||
enable_service() {
|
||||
command -v systemctl &>/dev/null || err "systemctl not found — cannot create the listener service"
|
||||
# Abort BEFORE writing the unit if the user systemd bus is unreachable
|
||||
# (SSH/headless) — no orphaned unit, remediation printed.
|
||||
ensure_user_bus
|
||||
mkdir -p "$USER_SYSTEMD_DIR"
|
||||
|
||||
local runner
|
||||
|
||||
@@ -171,6 +171,9 @@ cmd_start() {
|
||||
log "(dry-run) install unit $USER_SYSTEMD_DIR/$SERVICE"
|
||||
log "(dry-run) systemctl --user daemon-reload && enable --now $SERVICE"
|
||||
else
|
||||
# User-bus pre-flight BEFORE writing the unit — abort with remediation
|
||||
# (no orphaned unit) when the user systemd bus is unreachable.
|
||||
ensure_user_bus
|
||||
mkdir -p "$USER_SYSTEMD_DIR"
|
||||
cat > "$USER_SYSTEMD_DIR/$SERVICE" <<EOF
|
||||
[Unit]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# GEN:START posflags
|
||||
declare -A _pos_flags
|
||||
_pos_flags[ai-hf]="--branch --gguf --list --output --quant --include --exclude --revision"
|
||||
_pos_flags[ai-server]="--port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots"
|
||||
_pos_flags[ai-server]="--port --host --model --ctx --gpu --threads --gpu-layers --gpu-threads --tensor-split --n-gpu-layers --batch-size --ubatch-size --temperature --top-k --top-p --repetition-penalty --mmap --mlock --kv-cache --ctx-size --metrics --health --slots --no-unit"
|
||||
_pos_flags[communication-matrix-listener]="--enable --disable --status --run"
|
||||
_pos_flags[communication-telegram-listener]="--enable --disable --status --sync-commands --run"
|
||||
_pos_flags[communication-telegram-sender]="--type --caption --parse-mode --no-preview --token --chat-id --markdown"
|
||||
|
||||
@@ -139,6 +139,20 @@ confirm() {
|
||||
esac
|
||||
}
|
||||
|
||||
# ── User systemd bus pre-flight ────────────────────────────────
|
||||
# ensure_user_bus — verify the per-user systemd bus is reachable BEFORE a tool
|
||||
# writes a unit file or calls `systemctl --user`. SSH/headless shells often
|
||||
# have neither XDG_RUNTIME_DIR nor DBUS_SESSION_BUS_ADDRESS set; systemctl
|
||||
# then fails AFTER an orphaned unit was already written. On failure errs with
|
||||
# remediation and never returns, so callers abort before writing anything.
|
||||
ensure_user_bus() {
|
||||
systemctl --user show-environment &>/dev/null && return 0
|
||||
err "cannot reach the user systemd bus (common in SSH/headless sessions) — systemctl --user failed
|
||||
fix: export XDG_RUNTIME_DIR=/run/user/$(id -u) (if the directory exists)
|
||||
fix: sudo loginctl enable-linger $(id -un) (persist the user session so the bus survives logouts)
|
||||
no unit was written — fix the bus and retry"
|
||||
}
|
||||
|
||||
# ── system.env loader ──────────────────────────────────────────
|
||||
# Shared "system" tool config (~/.config/linux_post_install/system.env).
|
||||
# Fills only variables that are not already exported — an explicitly-set
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# t-ai-server-bus.sh — F4 + E2E regression for the user systemd-bus pre-flight.
|
||||
#
|
||||
# F4 contract:
|
||||
# - `pos ai server start` (unit path) calls ensure_user_bus BEFORE the unit
|
||||
# is written; with no user bus (SSH/headless: XDG_RUNTIME_DIR and
|
||||
# DBUS_SESSION_BUS_ADDRESS unset, systemctl --user fails) it errs with BOTH
|
||||
# remediation lines and leaves NO orphaned unit file.
|
||||
# - with a reachable bus the command proceeds past the pre-flight.
|
||||
# - `--no-unit` is the bus-less escape hatch: it skips the pre-flight, runs
|
||||
# llama-server directly under nohup with a pidfile, and stop/status can
|
||||
# find it via the pidfile.
|
||||
#
|
||||
# E2E regression (the reported user scenario):
|
||||
# - install-shaped stubs + working bus: `start Qwen-Qwen3-1.7B-GGUF` resolves
|
||||
# the model dir (F3), writes a correct unit (F6: all 5 flags + --port
|
||||
# 8088), reports a resolvable version (F1: 0.4.0, not "unknown"), and the
|
||||
# unit passes `systemd-analyze verify`.
|
||||
# - SSH-shaped no-bus env: clean pre-flight err, remediation lines, NO unit.
|
||||
|
||||
run_test() {
|
||||
local sandbox stubs models unitdir runtime
|
||||
sandbox="$(mksandbox ai-server-bus)"
|
||||
stubs="$sandbox/stubs"
|
||||
models="$sandbox/models"
|
||||
unitdir="$sandbox/userunits"
|
||||
runtime="$sandbox/runtime"
|
||||
mkdir -p "$stubs" "$models" "$unitdir" "$runtime"
|
||||
|
||||
# ── User's real HF layout (F3 fixture) ──
|
||||
mkdir -p "$models/Qwen-Qwen3-1.7B-GGUF"
|
||||
touch "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf"
|
||||
|
||||
# ── Stubs ──
|
||||
# llama-server: b10822-shaped (version → stderr, full-support help).
|
||||
cat > "$stubs/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
--version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;;
|
||||
--help)
|
||||
cat <<'HELP'
|
||||
usage: llama-server [options]
|
||||
options:
|
||||
--host <addr> bind address
|
||||
--port <port> server port
|
||||
--n-gpu-layers <n> layers to offload
|
||||
--ctx-size <n> context size
|
||||
--threads <n> cpu threads
|
||||
HELP
|
||||
;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
STUB
|
||||
# systemctl: FAILS exactly like the real bus-missing case (SSH).
|
||||
cat > "$stubs/systemctl" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
echo "Failed to connect to user scope bus via local transport: \$DBUS_SESSION_BUS_ADDRESS and \$XDG_RUNTIME_DIR not defined" >&2
|
||||
exit 1
|
||||
STUB
|
||||
# nvidia-smi: no GPU → deterministic CPU path.
|
||||
printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi"
|
||||
# curl: healthy JSON (health + models probes).
|
||||
printf '#!/usr/bin/env bash\nprintf "%%s" '"'"'{"status":"ok"}'"'"'\n' > "$stubs/curl"
|
||||
# loginctl: Linger=no (so the linger warn path is deterministic).
|
||||
printf '#!/usr/bin/env bash\nprintf "Linger=no\\n"\n' > "$stubs/loginctl"
|
||||
chmod +x "$stubs/llama-server" "$stubs/systemctl" "$stubs/nvidia-smi" "$stubs/curl" "$stubs/loginctl"
|
||||
|
||||
local server="$ROOT/bin/pos-ai-server"
|
||||
local nobus_env=(-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS
|
||||
PATH="$stubs:/usr/bin:/bin" HF_DOWNLOAD_DIR="$models"
|
||||
USER_SYSTEMD_DIR="$unitdir" NO_UNIT_PIDFILE="$sandbox/pos-ai-server.pid"
|
||||
NO_UNIT_LOG="$sandbox/pos-ai-server.log")
|
||||
local bus_env=(-u XDG_RUNTIME_DIR -u DBUS_SESSION_BUS_ADDRESS
|
||||
PATH="$stubs:/usr/bin:/bin" HF_DOWNLOAD_DIR="$models"
|
||||
USER_SYSTEMD_DIR="$unitdir" XDG_RUNTIME_DIR="$runtime"
|
||||
DBUS_SESSION_BUS_ADDRESS="unix:path=$runtime/bus"
|
||||
NO_UNIT_PIDFILE="$sandbox/pos-ai-server.pid" NO_UNIT_LOG="$sandbox/pos-ai-server.log")
|
||||
|
||||
# ═══ F4 pre-flight: bus missing → err + remediation + NO unit ═══
|
||||
test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF
|
||||
check_contains "F4 bus-missing start names the bus problem" "user systemd bus" "$TR_OUT"
|
||||
check_contains "F4 remediation: export XDG_RUNTIME_DIR" "export XDG_RUNTIME_DIR=/run/user/" "$TR_OUT"
|
||||
check_contains "F4 remediation: sudo loginctl enable-linger" "sudo loginctl enable-linger" "$TR_OUT"
|
||||
check_contains "F4 states no unit was written" "no unit was written" "$TR_OUT"
|
||||
[ "$TR_RC" -ne 0 ] && printf ' PASS F4 bus-missing start exits nonzero\n' || printf ' FAIL F4 bus-missing start exited 0\n'
|
||||
check_file_absent "F4 no orphaned unit written" "$unitdir/pos-ai-server.service"
|
||||
|
||||
# ═══ F4 pre-flight: bus present → proceeds past pre-flight (DRY_RUN) ═══
|
||||
# Replace the failing systemctl with a succeeding one.
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/systemctl"
|
||||
test_run_env "${bus_env[@]}" DRY_RUN=1 -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF
|
||||
check_rc "F4 bus-present DRY_RUN start exits 0" 0 "$TR_RC"
|
||||
check_contains "F4 bus-present proceeds past pre-flight (ExecStart)" "ExecStart:" "$TR_OUT"
|
||||
|
||||
# ═══ F4 --no-unit escape hatch (bus-less direct run) ═══
|
||||
# A real --no-unit start must skip the bus check and launch the stub under
|
||||
# nohup with a pidfile; status finds it; stop kills it. The stub sleeps via
|
||||
# `exec sleep` so the recorded pid IS the sleeper (no orphan children).
|
||||
cat > "$stubs/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
--version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;;
|
||||
--help)
|
||||
cat <<'HELP'
|
||||
usage: llama-server [options]
|
||||
options:
|
||||
--host <addr> bind address
|
||||
--port <port> server port
|
||||
--n-gpu-layers <n> layers to offload
|
||||
--ctx-size <n> context size
|
||||
--threads <n> cpu threads
|
||||
HELP
|
||||
;;
|
||||
*) exec sleep 300 ;;
|
||||
esac
|
||||
STUB
|
||||
chmod +x "$stubs/llama-server"
|
||||
|
||||
# failing systemctl stub again: --no-unit must NOT care.
|
||||
cat > "$stubs/systemctl" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
echo "no bus" >&2
|
||||
exit 1
|
||||
STUB
|
||||
chmod +x "$stubs/systemctl"
|
||||
|
||||
test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF --no-unit
|
||||
check_rc "F4 --no-unit bus-less start exits 0" 0 "$TR_RC"
|
||||
check_file_exists "F4 --no-unit writes pidfile" "$sandbox/pos-ai-server.pid"
|
||||
check_contains "F4 --no-unit prints log path hint" "log:" "$TR_OUT"
|
||||
local direct_pid
|
||||
direct_pid="$(cat "$sandbox/pos-ai-server.pid" 2>/dev/null || true)"
|
||||
if [[ "$direct_pid" =~ ^[0-9]+$ ]] && kill -0 "$direct_pid" 2>/dev/null; then
|
||||
printf ' PASS F4 --no-unit launched a live server process (pid %s)\n' "$direct_pid"
|
||||
else
|
||||
printf ' FAIL F4 --no-unit pid %s not alive\n' "${direct_pid:-EMPTY}"
|
||||
fi
|
||||
|
||||
# status via the pidfile reports running (deps + curl stubs present).
|
||||
test_run_env "${nobus_env[@]}" -- timeout 30 "$server" status
|
||||
check_rc "F4 status exits 0 after --no-unit start" 0 "$TR_RC"
|
||||
check_contains "F4 status finds direct-run server running" "service: running" "$TR_OUT"
|
||||
|
||||
# stop kills the pid and removes the pidfile.
|
||||
test_run_env "${nobus_env[@]}" -- timeout 30 "$server" stop
|
||||
check_rc "F4 --no-unit stop exits 0" 0 "$TR_RC"
|
||||
check_file_absent "F4 stop removes pidfile" "$sandbox/pos-ai-server.pid"
|
||||
if [[ "$direct_pid" =~ ^[0-9]+$ ]] && ! kill -0 "$direct_pid" 2>/dev/null; then
|
||||
printf ' PASS F4 stop killed the direct-run process\n'
|
||||
else
|
||||
printf ' FAIL F4 direct-run process still alive after stop\n'
|
||||
fi
|
||||
|
||||
# ═══ E2E positive: user scenario with a working bus ═══
|
||||
# Restore full-support stub + succeeding systemctl.
|
||||
cat > "$stubs/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
--version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;;
|
||||
--help)
|
||||
cat <<'HELP'
|
||||
usage: llama-server [options]
|
||||
options:
|
||||
--host <addr> bind address
|
||||
--port <port> server port
|
||||
--n-gpu-layers <n> layers to offload
|
||||
--ctx-size <n> context size
|
||||
--threads <n> cpu threads
|
||||
HELP
|
||||
;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
STUB
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/llama-server" "$stubs/systemctl"
|
||||
|
||||
test_run_env "${bus_env[@]}" -- timeout 60 "$server" start Qwen-Qwen3-1.7B-GGUF
|
||||
check_rc "E2E user scenario start exits 0" 0 "$TR_RC"
|
||||
check_file_exists "E2E unit written" "$unitdir/pos-ai-server.service"
|
||||
|
||||
local unit="$unitdir/pos-ai-server.service"
|
||||
local unit_exec
|
||||
unit_exec="$(grep '^ExecStart=' "$unit")"
|
||||
check_contains "E2E unit: model expanded from dir" "Qwen3-1.7B-Q8_0.gguf" "$unit_exec"
|
||||
check_contains "E2E unit: --port 8088 pinned" "--port 8088" "$unit_exec"
|
||||
check_contains "E2E unit: --host present" "--host 127.0.0.1" "$unit_exec"
|
||||
check_contains "E2E unit: --n-gpu-layers present" "--n-gpu-layers 0" "$unit_exec"
|
||||
check_contains "E2E unit: --ctx-size present" "--ctx-size 4096" "$unit_exec"
|
||||
check_contains "E2E unit: --threads present" "--threads" "$unit_exec"
|
||||
|
||||
# Version resolvable through the real CLI on the same run's stub.
|
||||
test_run_env "${bus_env[@]}" -- timeout 30 "$server" status
|
||||
check_contains "E2E version is resolvable (0.4.0)" "version: 0.4.0" "$TR_OUT"
|
||||
check_not_contains "E2E version is NOT unknown" "version: unknown" "$TR_OUT"
|
||||
|
||||
# Unit must pass systemd-analyze verify.
|
||||
if command -v systemd-analyze >/dev/null 2>&1; then
|
||||
test_run systemd-analyze verify "$unit"
|
||||
check_rc "E2E systemd-analyze verify accepts unit" 0 "$TR_RC"
|
||||
else
|
||||
skip_case "E2E systemd-analyze verify" "systemd-analyze not available"
|
||||
fi
|
||||
|
||||
# stop cleans up (unit removed).
|
||||
test_run_env "${bus_env[@]}" -- timeout 30 "$server" stop
|
||||
check_rc "E2E stop exits 0" 0 "$TR_RC"
|
||||
check_file_absent "E2E stop removes unit" "$unitdir/pos-ai-server.service"
|
||||
|
||||
# ═══ E2E SSH-shaped: no bus → clean pre-flight err, remediation, NO unit ═══
|
||||
cat > "$stubs/systemctl" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
echo "Failed to connect to user scope bus via local transport: \$DBUS_SESSION_BUS_ADDRESS and \$XDG_RUNTIME_DIR not defined" >&2
|
||||
exit 1
|
||||
STUB
|
||||
chmod +x "$stubs/systemctl"
|
||||
test_run_env "${nobus_env[@]}" -- timeout 30 "$server" start Qwen-Qwen3-1.7B-GGUF
|
||||
check_rc "E2E SSH-shaped start exits nonzero" 1 "$TR_RC"
|
||||
check_contains "E2E SSH-shaped: bus problem named" "user systemd bus" "$TR_OUT"
|
||||
check_contains "E2E SSH-shaped: export XDG_RUNTIME_DIR remediation" "export XDG_RUNTIME_DIR=/run/user/" "$TR_OUT"
|
||||
check_contains "E2E SSH-shaped: loginctl enable-linger remediation" "sudo loginctl enable-linger" "$TR_OUT"
|
||||
check_contains "E2E SSH-shaped: no unit was written" "no unit was written" "$TR_OUT"
|
||||
check_file_absent "E2E SSH-shaped: no orphaned unit" "$unitdir/pos-ai-server.service"
|
||||
}
|
||||
@@ -40,7 +40,11 @@ STUB
|
||||
#!/usr/bin/env bash
|
||||
exit 1
|
||||
STUB
|
||||
chmod +x "$stubs/llama-server" "$stubs/nvidia-smi"
|
||||
# systemctl: succeed — F4's ensure_user_bus pre-flight must pass so the
|
||||
# dry-run reaches the ExecStart output this test asserts on (bus-missing
|
||||
# behavior is covered separately in t-ai-server-bus.sh).
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" "$stubs/systemctl"
|
||||
|
||||
local server="$ROOT/bin/pos-ai-server"
|
||||
local base_env=(PATH="$stubs:/usr/bin:/bin" DRY_RUN=1
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# t-ai-server-model.sh — F3: `resolve_model` directory expansion regression.
|
||||
# The HF downloader writes weights as `$HF_DOWNLOAD_DIR/<repo-slug>/<file>.gguf`,
|
||||
# but the old resolve_model accepted only FILE paths, so the user's
|
||||
# `pos ai server start Qwen-Qwen3-1.7B-GGUF` failed with "Model not found".
|
||||
# F3 adds: dir with exactly one .gguf → resolve; multiple → err listing each
|
||||
# as `<dir>/<file>` + "pick one"; zero → fall through to not-found. It also
|
||||
# accepts the `<name>/<file>.gguf` slug/file form. Flat-file + absolute paths
|
||||
# must keep working (backward compatibility).
|
||||
#
|
||||
# Unit level: call the REAL shipped resolve_model / resolve_gguf_in_dir with a
|
||||
# sandbox HF_DOWNLOAD_DIR. Integration level: `start` DRY_RUN resolves the
|
||||
# user's real dir name.
|
||||
|
||||
# extract_fn <source-file> <fnname> — print one brace-delimited function body.
|
||||
extract_fn() {
|
||||
local file="$1" fn="$2"
|
||||
awk -v fn="$fn" '
|
||||
BEGIN { found=0; depth=0 }
|
||||
{
|
||||
if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 }
|
||||
if (found) {
|
||||
n_open = gsub(/\{/, "{")
|
||||
n_close = gsub(/\}/, "}")
|
||||
depth = depth + n_open - n_close
|
||||
print
|
||||
if (depth <= 0) exit
|
||||
}
|
||||
}
|
||||
' "$file"
|
||||
}
|
||||
|
||||
run_test() {
|
||||
source "$ROOT/lib/common.sh" # err/log for resolve_model
|
||||
|
||||
local sandbox models aid
|
||||
sandbox="$(mksandbox ai-server-model)"
|
||||
models="$sandbox/models"
|
||||
mkdir -p "$models"
|
||||
aid="$ROOT/bin/pos-ai-server"
|
||||
|
||||
# Extract the REAL model-resolution functions.
|
||||
local fn_file="$sandbox/fns.sh" com="$ROOT/lib/common.sh"
|
||||
{
|
||||
extract_fn "$aid" resolve_gguf_in_dir
|
||||
extract_fn "$aid" resolve_model
|
||||
} > "$fn_file"
|
||||
|
||||
# ── Fixtures (mirror the user's real layout) ──
|
||||
# single-gguf dir (the reported case): Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf
|
||||
mkdir -p "$models/Qwen-Qwen3-1.7B-GGUF"
|
||||
touch "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf"
|
||||
# multi-gguf dir
|
||||
mkdir -p "$models/multi"
|
||||
touch "$models/multi/q1.gguf" "$models/multi/q2.gguf"
|
||||
# empty dir
|
||||
mkdir -p "$models/empty"
|
||||
# flat-file legacy form
|
||||
touch "$models/flat.gguf"
|
||||
# absolute file
|
||||
touch "$models/abs.gguf"
|
||||
|
||||
# ── F3 unit cases (direct function calls) ──
|
||||
# 1. single-gguf dir resolves to the inner file
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'Qwen-Qwen3-1.7B-GGUF'"
|
||||
check_rc "F3 single-gguf dir resolves (rc 0)" 0 "$TR_RC"
|
||||
check_eq "F3 resolves to inner gguf" "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" "$TR_OUT"
|
||||
|
||||
# 2. multi-gguf dir errors listing each as <dir>/<file> + "pick one"
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'multi'"
|
||||
check_rc "F3 multi-gguf dir errors (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F3 multi lists q1" "multi/q1.gguf" "$TR_OUT"
|
||||
check_contains "F3 multi lists q2" "multi/q2.gguf" "$TR_OUT"
|
||||
check_contains "F3 multi says pick one" "pick one" "$TR_OUT"
|
||||
|
||||
# 3. zero-gguf dir → falls through to today's not-found
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'empty'"
|
||||
check_rc "F3 empty dir errors (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F3 empty dir -> Model not found" "Model not found" "$TR_OUT"
|
||||
|
||||
# 4. slug/file form <name>/<file>.gguf resolves
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf'"
|
||||
check_rc "F3 slug/file form resolves (rc 0)" 0 "$TR_RC"
|
||||
check_eq "F3 slug/file resolves to inner gguf" "$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf" "$TR_OUT"
|
||||
|
||||
# 5. flat-file legacy form still works
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model 'flat.gguf'"
|
||||
check_rc "F3 flat file resolves (rc 0)" 0 "$TR_RC"
|
||||
check_eq "F3 flat file stays flat (backward compat)" "$models/flat.gguf" "$TR_OUT"
|
||||
|
||||
# 6. absolute file path still works
|
||||
test_run_env HF_DOWNLOAD_DIR="$models" -- bash -c "source '$com'; source '$fn_file'; resolve_model '$models/abs.gguf'"
|
||||
check_rc "F3 absolute file resolves (rc 0)" 0 "$TR_RC"
|
||||
check_eq "F3 absolute file unchanged" "$models/abs.gguf" "$TR_OUT"
|
||||
|
||||
# ── F3 integration: `start <dir>` DRY_RUN resolves through the full CLI ──
|
||||
local stubs="$sandbox/stubs"
|
||||
mkdir -p "$stubs"
|
||||
cat > "$stubs/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
--version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;;
|
||||
--help)
|
||||
cat <<'HELP'
|
||||
usage: llama-server [options]
|
||||
options:
|
||||
--host <addr> bind address
|
||||
--port <port> server port
|
||||
--n-gpu-layers <n> layers to offload
|
||||
--ctx-size <n> context size
|
||||
--threads <n> cpu threads
|
||||
HELP
|
||||
;;
|
||||
esac
|
||||
STUB
|
||||
printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi"
|
||||
chmod +x "$stubs/llama-server" "$stubs/nvidia-smi"
|
||||
|
||||
local server="$ROOT/bin/pos-ai-server"
|
||||
test_run_env PATH="$stubs:/usr/bin:/bin" DRY_RUN=1 \
|
||||
HF_DOWNLOAD_DIR="$models" USER_SYSTEMD_DIR="$sandbox/userunits" -- \
|
||||
"$server" start Qwen-Qwen3-1.7B-GGUF --no-unit
|
||||
check_rc "F3 CLI start with dir arg exits 0" 0 "$TR_RC"
|
||||
check_contains "F3 CLI resolves dir to inner file" \
|
||||
"-m \"$models/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf\"" "$TR_OUT"
|
||||
|
||||
# multi-gguf via CLI → clean err naming pick-one, rc 1
|
||||
test_run_env PATH="$stubs:/usr/bin:/bin" DRY_RUN=1 \
|
||||
HF_DOWNLOAD_DIR="$models" USER_SYSTEMD_DIR="$sandbox/userunits" -- \
|
||||
"$server" start multi --no-unit
|
||||
check_rc "F3 CLI multi-gguf start errors (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F3 CLI multi-gguf says pick one" "pick one" "$TR_OUT"
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# t-ai-server-validate.sh — F1/F2/F5/F6 regression for `pos ai server`:
|
||||
# F1 detect_llama_version reads llama.cpp's version string from STDERR
|
||||
# (real builds print `version: 0.4.0-dev (build 10822, …)` to stderr),
|
||||
# and falls back to "unknown" (no errexit) when unreadable/missing.
|
||||
# F2 flag validation is pipe-less (no printf|grep -q SIGPIPE/rc=141 race):
|
||||
# validate_default_flags / validate_requested_flags are deterministic —
|
||||
# all 5 defaults kept every run on a real-shaped ~59 KB help, and a
|
||||
# requested flag is always accepted/rejected the same way.
|
||||
# F5 find_llamacpp no longer matches an unrelated bare `server` binary.
|
||||
# F6 --port $PORT (and the 4 other defaults) always appear in ExecStart.
|
||||
#
|
||||
# Strategy: F1/F5/F2-internals call the REAL shipped functions directly
|
||||
# (brace-matching extraction, so production logic is never re-typed); F6
|
||||
# exercises the full CLI with the real-shaped help fixture.
|
||||
|
||||
# extract_fn <source-file> <fnname> — print one brace-delimited function body.
|
||||
# Used to unit-test a self-contained function that lives in a thick script
|
||||
# whose dispatch we must not trigger.
|
||||
extract_fn() {
|
||||
local file="$1" fn="$2"
|
||||
awk -v fn="$fn" '
|
||||
BEGIN { found=0; depth=0 }
|
||||
{
|
||||
if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 }
|
||||
if (found) {
|
||||
n_open = gsub(/\{/, "{")
|
||||
n_close = gsub(/\}/, "}")
|
||||
depth = depth + n_open - n_close
|
||||
print
|
||||
if (depth <= 0) exit
|
||||
}
|
||||
}
|
||||
' "$file"
|
||||
}
|
||||
|
||||
# ── F1/F2 help fixture: 732-line ~59 KB body carrying the 5 defaults at the
|
||||
# REAL llama.cpp line offsets (7/25/140/503/506) — the shape that used to
|
||||
# trip the printf|grep -q SIGPIPE race under pipefail.
|
||||
build_real_help() {
|
||||
local total=732 i
|
||||
for ((i = 1; i <= total; i++)); do
|
||||
case "$i" in
|
||||
7) printf '%s\n' ' -t, --threads N' ;;
|
||||
25) printf '%s\n' ' -c, --ctx-size N' ;;
|
||||
140) printf '%s\n' ' -ngl, --gpu-layers, --n-gpu-layers N' ;;
|
||||
503) printf '%s\n' '--host HOST' ;;
|
||||
506) printf '%s\n' '--port PORT' ;;
|
||||
*)
|
||||
if [ "$i" -le 2 ]; then
|
||||
printf 'usage: llama-server [options]\n'
|
||||
else
|
||||
printf ' --general-option-%04d <arg> example option number %d with a longish description\n' "$i" "$i"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
new_stub() { # new_stub <dir> — full-support llama-server stub (stderr version)
|
||||
local dir="$1" help_fixture
|
||||
help_fixture="$(build_real_help)"
|
||||
cat > "$dir/llama-server" <<STUB
|
||||
#!/usr/bin/env bash
|
||||
case "\$1" in
|
||||
--version) printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2 ;;
|
||||
--help)
|
||||
cat <<'EOF_HELP'
|
||||
$help_fixture
|
||||
EOF_HELP
|
||||
;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
STUB
|
||||
chmod +x "$dir/llama-server"
|
||||
}
|
||||
|
||||
run_test() {
|
||||
source "$ROOT/lib/common.sh" # err/warn/log for the extracted functions
|
||||
|
||||
local sandbox stubs
|
||||
sandbox="$(mksandbox ai-server-validate)"
|
||||
stubs="$sandbox/stubs"
|
||||
mkdir -p "$stubs"
|
||||
|
||||
local aid="$ROOT/bin/pos-ai-server"
|
||||
local fn_file="$sandbox/fns.sh"
|
||||
|
||||
# ═══ F1: detect_llama_version (STDERR capture + "unknown" guard) ═══
|
||||
extract_fn "$aid" detect_llama_version > "$fn_file"
|
||||
source "$fn_file"
|
||||
|
||||
cat > "$stubs/f1-stderr" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2
|
||||
STUB
|
||||
chmod +x "$stubs/f1-stderr"
|
||||
test_run detect_llama_version "$stubs/f1-stderr"
|
||||
check_rc "F1 stderr semver -> rc 0" 0 "$TR_RC"
|
||||
check_eq "F1 stderr semver resolved (not unknown)" "0.4.0" "$TR_OUT"
|
||||
|
||||
cat > "$stubs/f1-build" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf 'version: build 10822\n' >&2
|
||||
STUB
|
||||
chmod +x "$stubs/f1-build"
|
||||
test_run detect_llama_version "$stubs/f1-build"
|
||||
check_rc "F1 build-only -> rc 0" 0 "$TR_RC"
|
||||
check_eq "F1 build 10822 -> 10822" "10822" "$TR_OUT"
|
||||
|
||||
cat > "$stubs/f1-b" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf 'version: b10822\n' >&2
|
||||
STUB
|
||||
chmod +x "$stubs/f1-b"
|
||||
test_run detect_llama_version "$stubs/f1-b"
|
||||
check_eq "F1 b10822 -> b10822" "b10822" "$TR_OUT"
|
||||
|
||||
# missing binary → "unknown", rc 0, NO errexit.
|
||||
test_run detect_llama_version "$sandbox/does-not-exist-llama-server"
|
||||
check_rc "F1 missing binary -> rc 0 (no errexit)" 0 "$TR_RC"
|
||||
check_eq "F1 missing binary -> unknown" "unknown" "$TR_OUT"
|
||||
|
||||
# binary exists but --version prints nothing parseable → unknown.
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/f1-noop"
|
||||
chmod +x "$stubs/f1-noop"
|
||||
test_run detect_llama_version "$stubs/f1-noop"
|
||||
check_eq "F1 unreadable version -> unknown" "unknown" "$TR_OUT"
|
||||
|
||||
# ═══ F5: find_llamacpp candidate narrowing ═══
|
||||
extract_fn "$aid" find_llamacpp > "$fn_file"
|
||||
source "$fn_file"
|
||||
|
||||
local onlyserver="$sandbox/onlyserver"
|
||||
mkdir -p "$onlyserver"
|
||||
ln -s /usr/bin/false "$onlyserver/server"
|
||||
(
|
||||
PATH="$onlyserver:/usr/bin:/bin"
|
||||
test_run find_llamacpp
|
||||
printf '%s' "$TR_RC" > "$sandbox/f5.rc"
|
||||
printf '%s' "$TR_OUT" > "$sandbox/f5.out"
|
||||
)
|
||||
check_rc "F5 bare 'server' alone NOT found (rc != 0)" 1 "$(cat "$sandbox/f5.rc")"
|
||||
check_eq "F5 'server' candidate yields nothing" "" "$(cat "$sandbox/f5.out")"
|
||||
|
||||
# llama-server IS a legitimate candidate; found first (rc 0 name).
|
||||
local havebin="$sandbox/havebin"
|
||||
mkdir -p "$havebin"
|
||||
new_stub "$havebin"
|
||||
(
|
||||
PATH="$havebin:/usr/bin:/bin"
|
||||
test_run find_llamacpp
|
||||
printf '%s' "$TR_RC" > "$sandbox/f5b.rc"
|
||||
printf '%s' "$TR_OUT" > "$sandbox/f5b.out"
|
||||
)
|
||||
check_rc "F5 llama-server present -> found (rc 0)" 0 "$(cat "$sandbox/f5b.rc")"
|
||||
check_eq "F5 returns llama-server (not server)" "llama-server" "$(cat "$sandbox/f5b.out")"
|
||||
|
||||
# ═══ F2: deterministic flag validation (pipe-less, no rc=141) ═══
|
||||
# Extract both validation functions; they use err/warn from common.sh.
|
||||
{
|
||||
extract_fn "$aid" validate_requested_flags
|
||||
extract_fn "$aid" validate_default_flags
|
||||
} > "$fn_file"
|
||||
local com="$ROOT/lib/common.sh"
|
||||
|
||||
# Default-flag determinism: 25 runs over the real-shaped help; every run
|
||||
# must keep all 5 defaults (the pre-fix SIGPIPE race dropped a random
|
||||
# subset per run). Run in a subshell sourcing the exact function bodies +
|
||||
# common.sh, capturing the globals it sets.
|
||||
local run stable=1 gvals run_rc
|
||||
for run in $(seq 1 25); do
|
||||
gvals="$(bash -c "
|
||||
source '$com'
|
||||
source '$fn_file'
|
||||
validate_default_flags '$havebin/llama-server' 0.4.0 ''
|
||||
printf '%s|%s|%s|%s|%s' \"\${DEFAULT_PORT_OK}\" \"\${DEFAULT_HOST_OK}\" \\
|
||||
\"\${DEFAULT_GPU_OK}\" \"\${DEFAULT_CTX_OK}\" \"\${DEFAULT_THREADS_OK}\"
|
||||
" 2>/dev/null)"
|
||||
run_rc=$?
|
||||
if [ "$run_rc" -ne 0 ]; then
|
||||
stable=0
|
||||
printf ' FAIL F2 default-flag validation run %s: whole invocation rc=%s (possible rc=141 SIGPIPE component)\n' "$run" "$run_rc"
|
||||
break
|
||||
fi
|
||||
if [ "$gvals" != "1|1|1|1|1" ]; then
|
||||
stable=0
|
||||
printf ' FAIL F2 run %s dropped a default (globals=%s)\n' "$run" "$gvals"
|
||||
break
|
||||
fi
|
||||
done
|
||||
[ "$stable" -eq 1 ] && printf ' PASS F2 validate_default_flags keeps all 5 defaults across 25 pipefail runs (whole-invocation rc=0 every run)\n'
|
||||
|
||||
# Requested-flag determinism (20+ runs each): supported always rc 0,
|
||||
# unsupported always hard-errors (never a flaky pass).
|
||||
local ok=0 bad=0
|
||||
for run in $(seq 1 20); do
|
||||
if bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --port" 2>/dev/null; then
|
||||
ok=$((ok + 1))
|
||||
fi
|
||||
if bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --tensor-split 1" 2>/dev/null; then
|
||||
bad=$((bad + 1))
|
||||
fi
|
||||
done
|
||||
check_eq "F2 requested supported --port accepted every run" 20 "$ok"
|
||||
check_eq "F2 no flaky pass on unsupported requested flag" 0 "$bad"
|
||||
|
||||
test_run bash -c "source '$com'; source '$fn_file'; validate_requested_flags '$havebin/llama-server' 0.4.0 --tensor-split 1"
|
||||
check_rc "F2 unsupported requested flag exits 1" 1 "$TR_RC"
|
||||
check_contains "F2 unsupported requested flag names it" "--tensor-split" "$TR_OUT"
|
||||
check_contains "F2 unsupported requested flag is version-aware" "0.4.0" "$TR_OUT"
|
||||
|
||||
# ═══ F6: full CLI — port pinning + version not "unknown" ═══
|
||||
local models="$sandbox/models"
|
||||
mkdir -p "$models"
|
||||
touch "$models/my-model.gguf"
|
||||
printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi"
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/nvidia-smi" "$stubs/systemctl"
|
||||
|
||||
local server="$ROOT/bin/pos-ai-server"
|
||||
local base_env=(PATH="$havebin:$stubs:/usr/bin:/bin" DRY_RUN=1
|
||||
USER_SYSTEMD_DIR="$sandbox/userunits")
|
||||
test_run_env "${base_env[@]}" -- "$server" start "$models/my-model.gguf"
|
||||
check_rc "F6 DRY_RUN start exits 0" 0 "$TR_RC"
|
||||
local exec_line
|
||||
exec_line="$(printf '%s\n' "$TR_OUT" | grep 'dry-run) ExecStart:' | sed 's/.*ExecStart: //')"
|
||||
check_contains "F6 --port 8088 pinned in ExecStart" "--port 8088" "$exec_line"
|
||||
check_contains "F6 --host present" "--host 127.0.0.1" "$exec_line"
|
||||
check_contains "F6 --n-gpu-layers present" "--n-gpu-layers 0" "$exec_line"
|
||||
check_contains "F6 --ctx-size present" "--ctx-size 4096" "$exec_line"
|
||||
check_contains "F6 --threads present" "--threads" "$exec_line"
|
||||
|
||||
# Version via the same b10822-shaped stub: status shows 0.4.0, never
|
||||
# "unknown" (F1 proven through the real CLI on a real-shaped build).
|
||||
test_run_env "${base_env[@]}" -- "$server" status
|
||||
check_rc "F1 CLI status exits 0" 0 "$TR_RC"
|
||||
check_contains "F1 CLI status resolves version 0.4.0" "version: 0.4.0" "$TR_OUT"
|
||||
check_not_contains "F1 CLI status version is NOT unknown" "version: unknown" "$TR_OUT"
|
||||
}
|
||||
@@ -122,7 +122,11 @@ HELP
|
||||
esac
|
||||
STUB
|
||||
printf '#!/usr/bin/env bash\nexit 1\n' > "$stubs/nvidia-smi"
|
||||
chmod +x "$stubs/llama-server" "$stubs/nvidia-smi"
|
||||
# systemctl: succeed — F4's ensure_user_bus pre-flight must pass so Part B's
|
||||
# dry-run reaches the ExecStart output it asserts on (bus-missing behavior
|
||||
# lives in t-ai-server-bus.sh, not here).
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/llama-server" "$stubs/nvidia-smi" "$stubs/systemctl"
|
||||
|
||||
local models="$sandbox/models"
|
||||
mkdir -p "$models"
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# t-llamacpp-install.sh — F7: `apps/ai/llamacpp.sh` post-install sanity.
|
||||
# After the installer drops llama-server + symlinks it into the bin dir, the
|
||||
# sanity check must verify the binary actually runs (version/help readable via
|
||||
# the same stdout/stderr the tool chain reads) and the symlink is not dangling.
|
||||
# healthy → passes, logs "llama.cpp sanity OK"
|
||||
# dangling symlink → err (rc 1, "dangling symlink")
|
||||
# non-executable → err (rc 1, not on PATH)
|
||||
# broken binary → err (rc 1, --version did not run → missing shared lib)
|
||||
# Uses the LLAMACPP_BIN_DIR seam; calls the REAL shipped function.
|
||||
#
|
||||
# Note: the real installer cannot run in the sandbox (network + sudo + root
|
||||
# paths), so the sanity function is extracted verbatim and exercised directly.
|
||||
|
||||
# extract_fn <source-file> <fnname> — print one brace-delimited function body.
|
||||
extract_fn() {
|
||||
local file="$1" fn="$2"
|
||||
awk -v fn="$fn" '
|
||||
BEGIN { found=0; depth=0 }
|
||||
{
|
||||
if (!found && $0 ~ ("^" fn "\\(\\)")) { found=1; depth=0 }
|
||||
if (found) {
|
||||
n_open = gsub(/\{/, "{")
|
||||
n_close = gsub(/\}/, "}")
|
||||
depth = depth + n_open - n_close
|
||||
print
|
||||
if (depth <= 0) exit
|
||||
}
|
||||
}
|
||||
' "$file"
|
||||
}
|
||||
|
||||
run_test() {
|
||||
local app="$ROOT/apps/ai/llamacpp.sh"
|
||||
local com="$ROOT/lib/common.sh"
|
||||
|
||||
# Extract the REAL llamacpp_sanity from the shipped installer.
|
||||
local fn_file="$TEST_TMP/llamacpp-fns.sh"
|
||||
extract_fn "$app" llamacpp_sanity > "$fn_file"
|
||||
|
||||
local sandbox bindir
|
||||
sandbox="$(mksandbox llamacpp-install)"
|
||||
bindir="$sandbox/bin"
|
||||
mkdir -p "$bindir"
|
||||
|
||||
# 1. healthy fixture → rc 0 + "llama.cpp sanity OK"
|
||||
cat > "$bindir/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
# healthy: version + help both readable; llama.cpp prints version to stderr
|
||||
printf 'version: 0.4.0-dev (build 10822, commit c457e3bf7)\n' >&2
|
||||
exit 0
|
||||
STUB
|
||||
chmod +x "$bindir/llama-server"
|
||||
test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \
|
||||
bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1
|
||||
check_rc "F7 healthy fixture sanity rc 0" 0 "$TR_RC"
|
||||
check_contains "F7 healthy fixture sanity OK log" "llama.cpp sanity OK" "$TR_OUT"
|
||||
|
||||
# 2. dangling symlink → err, rc 1, names the broken link
|
||||
rm -f "$bindir/llama-server"
|
||||
ln -s "$bindir/no-such-target" "$bindir/llama-server"
|
||||
test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \
|
||||
bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1
|
||||
check_rc "F7 dangling symlink errs (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F7 dangling symlink message" "dangling symlink" "$TR_OUT"
|
||||
|
||||
# 3. non-executable fixture → err, rc 1 (bash `command -v` finds the file
|
||||
# by PATH existence; the binary fails when actually executed, so the
|
||||
# sanity errs at the --version step — the contract is a non-zero err).
|
||||
rm -f "$bindir/llama-server"
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$bindir/llama-server"
|
||||
chmod 644 "$bindir/llama-server" # NOT executable
|
||||
test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \
|
||||
bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1
|
||||
check_rc "F7 non-executable errs (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F7 non-executable message is a sanity err" "llama.cpp install sanity failed" "$TR_OUT"
|
||||
|
||||
# 4. broken binary (--version fails like a missing shared lib) → err, rc 1
|
||||
rm -f "$bindir/llama-server"
|
||||
cat > "$bindir/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
# simulates a truncated/missing-shared-lib binary: version unreadable
|
||||
echo "error while loading shared libraries: libllama.so: cannot open" >&2
|
||||
exit 1
|
||||
STUB
|
||||
chmod +x "$bindir/llama-server"
|
||||
test_run_env LLAMACPP_BIN_DIR="$bindir" PATH="$bindir:/usr/bin:/bin" -- \
|
||||
bash -c "source '$com'; source '$fn_file'; llamacpp_sanity" 2>&1
|
||||
check_rc "F7 broken binary errs (rc 1)" 1 "$TR_RC"
|
||||
check_contains "F7 broken binary message" "did not run" "$TR_OUT"
|
||||
|
||||
# 5. static guard: the shipped installer actually CALLS the sanity after
|
||||
# the symlink loop (the wiring that makes F7 real).
|
||||
if grep -q 'llamacpp_sanity' "$app"; then
|
||||
printf ' PASS F7 installer wires llamacpp_sanity into install\n'
|
||||
else
|
||||
printf ' FAIL F7 installer does not call llamacpp_sanity\n'
|
||||
fi
|
||||
}
|
||||
@@ -43,6 +43,11 @@ HELP
|
||||
esac
|
||||
STUB
|
||||
chmod +x "$stubs/llama-server"
|
||||
# systemctl: succeed — F4's ensure_user_bus pre-flight must pass so these
|
||||
# flag-validation checks reach their intended outcome (unsupported-flag /
|
||||
# default-flag behavior). Bus-missing behavior lives in t-ai-server-bus.sh.
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$stubs/systemctl"
|
||||
chmod +x "$stubs/systemctl"
|
||||
|
||||
# 1. CLI-explicit unsupported flag → rc 1, names flag + model version
|
||||
test_run_env "${base_env[@]}" -- "$server" start "$models/my-model.gguf" --tensor-split 1:2:3
|
||||
|
||||
Reference in New Issue
Block a user