admin/Linux_post_install
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

ai need continue
gates / consistency-and-conventions (push) Successful in 1m35s
Details

Browse Source
This commit is contained in:
he
2026-09-04 13:58:38 -04:00
parent adf88cc737
commit 6a6c323a89
10 changed files with 2007 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AgentsReport/architect/2026-09-04_llamacpp-server-architecture.md
+533
View File
@@ -0,0 +1,533 @@
# Architecture Decision: `pos ai server` — llama.cpp Inference Server
## TL;DR
**Decisions:**
1. New tool `bin/pos-ai-server` with subcommands: `start`, `stop`, `status`, `models`, `logs`
2. Systemd user service generated at runtime (same pattern as `pos-network-download` and `pos-communication-telegram-listener`)
3. Config extends existing `ai` scope in `ai.env` — no new config files
4. Provider adapter `lib/ai-providers/llamacpp.sh` follows the 4-function contract
5. Changes to `bin/pos-ai`: `resolve_key()` accepts `llamacpp` (no key needed), `resolve_model()` falls through to `LLAMACPP_MODEL`, `cmd_providers()` includes llamacpp
6. No static `systemd/` unit file — the service is generated dynamically because model path, port, and GPU flags are user-configurable
**Open items:**
- llama-server binary name varies by build (`llama-server`, `llama.cpp/server`, `server`) — detection logic needs a fallback chain
- ROCm detection deferred (Debian/Ubuntu focus, CUDA-only auto-detect)
---
## Decision 1: Tool Structure
**Problem:** User needs to start/stop/manage a local llama.cpp inference server via `pos`.
**Decision:** Create `bin/pos-ai-server` as a standalone tool under the `ai` category, with subcommands.
**Evidence:**
- Existing pattern: `bin/pos-network-download` is a standalone tool with `start`/`stop`/`status` subcommands for the aria2 daemon
- Existing pattern: `bin/pos-communication-telegram-listener` manages its own systemd user service
- Tool naming: `pos-ai-server` → `pos ai server` (category: `ai`, command: `server`)
**Subcommands:**
| Subcommand | Description |
|------------|-------------|
| `start [model]` | Install & start the systemd user service (model from arg, config, or interactive pick) |
| `stop` | Stop & remove the service |
| `status` | Show running state, loaded model, port, health endpoint |
| `models` | List available GGUF files from `HF_DOWNLOAD_DIR` |
| `logs [lines]` | Show recent server logs via `journalctl --user` |
**Not interactive:** `pos ai server` does NOT read stdin (no prompts that block under `tee`). It does NOT need to be in `INTERACTIVE_CMDS`.
**Approved scope:**
- `bin/pos-ai-server` — 1 file
- POS header: `# POS: ai server — llama.cpp local inference server (start, stop, status, models, logs)`
- POS_SUBCMDS: `start stop status models logs`
- POS_DEPS: `curl jq` (curl for health check + API, jq for JSON parsing)
- POS_FLAGS: `--port --host --model --ctx --gpu --threads`
[DECIDED]
---
## Decision 2: Systemd User Service (Runtime-Generated)
**Problem:** The llama-server service needs model path, port, GPU layers, and other parameters that are user-configurable. A static unit file can't carry these.
**Decision:** Generate the systemd user service file at runtime (same pattern as `pos-network-download` lines 172-187 and `pos-communication-telegram-listener` lines 481-500).
**Evidence:**
- `pos-network-download`: generates `pos-aria2.service` at `cmd_start()` with `$RPC_PORT`, `$RPC_SECRET`, `$DOWNLOAD_DIR` baked into `ExecStart`
- `pos-communication-telegram-listener`: generates `pos-telegram-listener.service` with the runner path baked in
- Both write to `$USER_SYSTEMD_DIR` (`~/.config/systemd/user/`), then `systemctl --user daemon-reload && enable --now`
- Both use `cat > "$USER_SYSTEMD_DIR/$SERVICE" <<EOF` pattern
**Service name:** `pos-ai-server.service`
**Service content:**
```ini
[Unit]
Description=pos llama.cpp inference server (linux-post-install)
After=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/env llama-server -m <MODEL> --port <PORT> --host <HOST> --n-gpu-layers <GPU_LAYERS> --ctx-size <CTX_SIZE> --threads <THREADS>
Restart=on-failure
RestartSec=5
TimeoutStopSec=10
KillMode=control-group
EnvironmentFile=-%h/.config/linux_post_install/ai.env
[Install]
WantedBy=default.target
```
**Key design choices:**
- `Type=simple` — llama-server runs in foreground by default (no daemonize flag needed)
- `Restart=on-failure` — restart if it crashes, but not on clean exit (`stop` sends SIGTERM, which is clean)
- `RestartSec=5` — give time for model unload/reload
- `TimeoutStopSec=10` — llama-server handles SIGTERM gracefully (unloads model), 10s is generous
- `KillMode=control-group` — ensures the whole process tree is cleaned up
- `EnvironmentFile=-` (dash prefix) — missing file is not an error
- ExecStart is a direct `llama-server` call (not a wrapper script) — systemd handles the lifecycle
**Where config values come from:** `cmd_start()` reads the config file, resolves all values, then bakes them into the generated unit. The `EnvironmentFile` line in the unit is a fallback but the actual arguments are baked in at generation time. This matches the aria2 pattern exactly.
**`start` subcommand flow:**
1. Load config from `ai.env`
2. Resolve model (argument → `LLAMACPP_MODEL` → interactive pick)
3. Resolve port, host, ctx, gpu, threads (flag → config → default)
4. Validate model file exists
5. Auto-detect GPU if `LLAMACPP_GPU_LAYERS` is `-1`
6. Check port availability
7. Generate systemd unit file
8. `systemctl --user daemon-reload`
9. `systemctl --user enable --now pos-ai-server.service`
10. Wait briefly, then check health endpoint
**Linger warning:** Same as existing tools — warn if `loginctl enable-linger` is needed.
[DECIDED]
---
## Decision 3: Config Keys (Extend `ai` Scope)
**Problem:** Server settings need to be persisted alongside existing AI config.
**Decision:** Extend the existing `ai` scope in `ai.env`. No new config file.
**Evidence:**
- `ai.env` already holds `AI_PROVIDER`, `AI_GEMINI_API_KEY`, `HF_DOWNLOAD_DIR`, etc.
- The `# POS_CONFIG:` header on `bin/pos-ai` already declares the `ai` scope
- Adding `LLAMACPP_*` keys to the same file keeps all AI config in one place
- `pos config ai` auto-discovers keys from `# POS_CONFIG:` headers
**Config keys to add:**
| Key | Default | Description |
|-----|---------|-------------|
| `LLAMACPP_PORT` | `8088` | Server listen port |
| `LLAMACPP_HOST` | `127.0.0.1` | Bind address |
| `LLAMACPP_MODEL` | *(empty)* | Default model path (GGUF file) |
| `LLAMACPP_CTX_SIZE` | `4096` | Context window size |
| `LLAMACPP_GPU_LAYERS` | `-1` | GPU layers (`-1` = auto-detect, `0` = CPU only) |
| `LLAMACPP_THREADS` | `$(nproc)` | CPU threads |
**POS_CONFIG header on `pos-ai`:** Extend the existing `# POS_CONFIG:` line to include the new keys. The existing header already uses `ai | ai.env | ...` format — we append `LLAMACPP_*` entries.
**New header addition (appended to existing `# POS_CONFIG:` line):**
```
| LLAMACPP_PORT=:Server port (default 8088) | LLAMACPP_HOST=:Bind address (default 127.0.0.1) | LLAMACPP_MODEL=:Default model path (GGUF) | LLAMACPP_CTX_SIZE:num:Context window size (default 4096) | LLAMACPP_GPU_LAYERS:num:GPU layers (-1=auto, 0=CPU only, default -1) | LLAMACPP_THREADS:num:CPU threads (default: nproc)
```
**Config template update:** Add commented examples to `config/ai.env`.
[DECIDED]
---
## Decision 4: Provider Adapter
**Problem:** `pos ai ask` should work with the local llama.cpp server as a backend, just like gemini/openrouter.
**Decision:** Create `lib/ai-providers/llamacpp.sh` with the 4-function contract.
**Evidence:**
- `lib/ai-providers/gemini.sh` and `lib/ai-providers/openrouter.sh` both implement: `provider_name()`, `provider_default_model()`, `provider_generate()`, `provider_models_list()`
- `pos-ai` loads providers via `load_provider()` which sources `$PROVIDER_DIR/$p.sh`
- The provider adapter pattern is established and stable
**Function signatures:**
```bash
# provider_name → human-readable name
provider_name() { printf 'Local llama.cpp'; }
# provider_default_model → what's loaded on the server
provider_default_model() {
local port="${LLAMACPP_PORT:-8088}"
local model
model="$(curl -sf "http://127.0.0.1:$port/v1/models" 2>/dev/null | jq -r '.data[0].id // empty')"
[ -n "$model" ] && printf '%s' "$model" || printf '(no model loaded)'
}
# provider_generate($1=model, $2=messages JSON, $3=optional system prompt)
# → POST to /v1/chat/completions, stdout = response text
provider_generate() {
local model="$1" messages="$2" system="${3:-}" port="${LLAMACPP_PORT:-8088}"
local body resp code body_out
# Build messages array with optional system prompt
if [ -n "$system" ]; then
body="$(printf '%s' "$messages" | jq -c --arg s "$system" \
'[{role:"system",content:$s}] + .messages')"
else
body="$(printf '%s' "$messages" | jq -c '.messages')"
fi
body="$(printf '%s' "$body" | jq -nc --arg m "$model" --argjson msgs "$body" \
'{model:$m, messages:$msgs, stream:false}')"
resp="$(curl -sS -m 120 -X POST "http://127.0.0.1:$port/v1/chat/completions" \
-H "Content-Type: application/json" \
--write-out $'\n%{http_code}' \
--data "$body")" || { echo "request failed (curl exit $?)" >&2; return 1; }
code="${resp##*$'\n'}"
body_out="${resp%$'\n'*}"
if [ "$code" != "200" ]; then
echo "API error $code" >&2
return 1
fi
printf '%s' "$body_out" | jq -r '.choices[0].message.content // ""'
}
# provider_models_list($1=current model) → stdout = formatted list
provider_models_list() {
local model="$1" port="${LLAMACPP_PORT:-8088}" resp code body
resp="$(curl -sf "http://127.0.0.1:$port/v1/models" \
--write-out $'\n%{http_code}')" || { echo "server not running" >&2; return 1; }
code="${resp##*$'\n'}"
body="${resp%$'\n'*}"
[ "$code" = "200" ] || { echo "API error $code" >&2; return 1; }
echo "Local llama.cpp models:"
printf '%s' "$body" | jq -r '.data[]? | .id' | while IFS= read -r m; do
[ -n "$m" ] || continue
if [ "$m" = "$model" ]; then
printf ' %-48s <- loaded\n' "$m"
else
printf ' %-48s\n' "$m"
fi
done
}
```
**No API key:** `resolve_key()` in `bin/pos-ai` needs a `llamacpp)` case that succeeds without a key. The local server has no auth.
**Provider detection in `resolve_key()`:**
```bash
llamacpp) return 0 ;; # No API key needed for local server
```
**Provider detection in `resolve_model()`:**
```bash
llamacpp)
[ -n "${LLAMACPP_MODEL:-}" ] && printf '%s' "$(basename "$LLAMACPP_MODEL")" && return ;;
```
**Provider detection in `cmd_providers()`:** The existing loop over `$PROVIDER_DIR/*.sh` auto-discovers `llamacpp.sh`. The `configured` check needs updating — llamacpp is "configured" when `llama-server` is available, not when an API key exists.
**POS_CONFIG header on adapter:** Add `# PROVIDER_CONFIG: LLAMACPP_MODEL=:Default model path (GGUF file)` to the adapter so `pos config ai` discovers it.
[DECIDED]
---
## Decision 5: GPU Auto-Detection
**Problem:** Auto-detect NVIDIA CUDA to set `--n-gpu-layers` appropriately.
**Decision:** Simple CUDA detection — check `nvidia-smi` and `/dev/nvidia*`. No ROCm for now (Debian/Ubuntu focus).
**Evidence:**
- `nvidia-smi` is the standard NVIDIA management interface
- `/dev/nvidia*` devices indicate driver presence
- llama.cpp uses `--n-gpu-layers -1` for auto (offload all possible layers to GPU)
- The user's request says "Debian/Ubuntu so mainly CUDA"
**Detection function (in `pos-ai-server`):**
```bash
detect_gpu() {
if command -v nvidia-smi &>/dev/null && nvidia-smi &>/dev/null 2>&1; then
echo "cuda"
else
echo "cpu"
fi
}
```
**GPU layer resolution:**
```bash
resolve_gpu_layers() {
local configured="${LLAMACPP_GPU_LAYERS:-}"
if [ -n "$configured" ] && [ "$configured" != "-1" ]; then
echo "$configured"
return
fi
# Auto-detect
local gpu
gpu="$(detect_gpu)"
case "$gpu" in
cuda) echo "-1" ;;
*) echo "0" ;;
esac
}
```
**Behavior:**
- `LLAMACPP_GPU_LAYERS=-1` (default) → auto-detect: CUDA → `-1`, no CUDA → `0`
- `LLAMACPP_GPU_LAYERS=0` → CPU only (override)
- `LLAMACPP_GPU_LAYERS=35` → explicit layer count (for fine-tuning)
**No ROCm:** Explicitly out of scope. The detection function can be extended later.
[DECIDED]
---
## Decision 6: Model Selection
**Problem:** User needs to pick from downloaded GGUF models.
**Decision:** `pos ai server models` scans `HF_DOWNLOAD_DIR` for `.gguf` files, reusing `pos ai hf list` patterns.
**Evidence:**
- `pos-ai-hf` downloads to `$HF_DOWNLOAD_DIR` (default `~/.local/share/linux_post_install/ai/models/`)
- `cmd_list()` in `pos-ai-hf` already scans for model directories with `.hf-meta` files
- GGUF files are the inference-ready format; they're the only files that matter for serving
**`models` subcommand behavior:**
```bash
cmd_models() {
local dir="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}"
[ -d "$dir" ] || { warn "No models directory — run 'pos ai hf download' first"; return 0; }
local found=0
echo "Available GGUF models:"
while IFS= read -r gguf; do
[ -f "$gguf" ] || continue
found=1
local name size
name="$(basename "$gguf")"
local dir_name
dir_name="$(basename "$(dirname "$gguf")")"
size="$(stat -c%s "$gguf" 2>/dev/null || echo 0)"
local human_size
human_size="$(human_size "$size")"
printf ' %-50s %s %s\n' "$dir_name/$name" "$human_size" ""
done < <(find "$dir" -name '*.gguf' -type f 2>/dev/null | sort)
[ "$found" -eq 0 ] && warn "No .gguf files found — download with 'pos ai hf download <repo> --gguf'"
}
```
**Model resolution order for `start [model]`:**
1. Explicit argument: `pos ai server start /path/to/model.gguf`
2. Relative path argument: `pos ai server start model.gguf` → search `HF_DOWNLOAD_DIR`
3. Config: `LLAMACPP_MODEL` from `ai.env`
4. Interactive pick: prompt user to select from available models
**Interactive pick (only when on a TTY and no model specified):**
```bash
pick_model() {
local models=() i
while IFS= read -r f; do
[ -f "$f" ] || continue
models+=("$f")
done < <(find "$HF_DOWNLOAD_DIR" -name '*.gguf' -type f 2>/dev/null | sort)
[ ${#models[@]} -gt 0 ] || err "No GGUF models found — run 'pos ai hf download <repo> --gguf'"
echo "Available models:"
for ((i = 0; i < ${#models[@]}; i++)); do
local name size
name="$(basename "${models[$i]}")"
size="$(stat -c%s "${models[$i]}" 2>/dev/null || echo 0)"
printf ' %2d) %-50s %s\n' "$((i + 1))" "$name" "$(human_size "$size")"
done
echo
local choice
printf 'Pick a model [1-%d]: ' "${#models[@]}"
IFS= read -r choice </dev/tty || choice=""
[[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "${#models[@]}" ] || err "Invalid selection"
printf '%s' "${models[$((choice - 1))]}"
}
```
[DECIDED]
---
## Decision 7: Health Check & Status
**Problem:** User needs to know if the server is running and healthy.
**Decision:** Use llama.cpp's `/health` endpoint + systemd state.
**`status` subcommand output:**
```
service: running
model: mistral-7b-v0.1.Q4_K_M.gguf
port: 8088
host: 127.0.0.1
gpu: CUDA (-1 layers)
context: 4096
threads: 16
autostart: enabled
endpoint: http://127.0.0.1:8088
health: ok (loaded)
```
**Health check function:**
```bash
check_health() {
local port="${LLAMACPP_PORT:-8088}"
local resp
resp="$(curl -sf "http://127.0.0.1:$port/health" 2>/dev/null)" || { echo "not running"; return 1; }
# llama.cpp /health returns {"status": "ok"} or {"status": "loading model", ...}
local status
status="$(printf '%s' "$resp" | jq -r '.status // "unknown"' 2>/dev/null)"
echo "$status"
}
```
**`logs` subcommand:** Uses `journalctl --user -u pos-ai-server -n <lines> --no-pager`.
[DECIDED]
---
## Decision 8: Changes to `bin/pos-ai`
**Problem:** `pos ai --provider llamacpp` should work, routing through the local server.
**Decision:** Minimal changes to `bin/pos-ai` — 3 touch points.
**Evidence:**
- `resolve_key()` has a `case "$p" in` that checks each provider — add `llamacpp)` case
- `resolve_model()` has a `case "$p" in` for provider-specific fallbacks — add `llamacpp)` case
- `cmd_providers()` checks API key configuration — add llamacpp case
- `require_key()` has provider-specific error messages — add llamacpp case
**Changes:**
1. **`resolve_key()` (line 167):** Add `llamacpp) return 0 ;;` — no key needed
2. **`resolve_model()` (line 193):** Add `llamacpp) [ -n "${LLAMACPP_MODEL:-}" ] && printf '%s' "$(basename "$LLAMACPP_MODEL")" && return ;;`
3. **`cmd_providers()` (line 120):** Add `llamacpp) [ -n "${LLAMACPP_PORT:-}" ] && configured="configured" || configured="configured (default port)" ;;` — local server is always "configured"
4. **`require_key()` (line 176):** Add `llamacpp) ;;` — no key needed, just return
These are all 1-2 line additions within existing `case` blocks.
[DECIDED]
---
## Decision 9: Error Handling
| Error | Detection | Response |
|-------|-----------|----------|
| `llama-server` not found | `command -v llama-server` fails | `err "llama-server not found — install llama.cpp (https://github.com/ggerganov/llama.cpp)"` |
| Port in use | `ss -tlnp` or `curl` to port | `err "Port $PORT already in use — check with 'ss -tlnp'"` |
| Model not found | `[ -f "$model" ]` | `err "Model not found: $model"` |
| GPU not detected | `detect_gpu` returns `cpu` | `warn "No NVIDIA GPU detected — running in CPU mode"` (continues) |
| Service start fails | `systemctl --user start` returns non-zero | `journalctl --user -u pos-ai-server -n 20 --no-pager` |
| Server unhealthy | `/health` returns non-200 or times out | `warn "Server may not be ready yet — check with 'pos ai server status'"` |
| Model too large | Not reliably detectable pre-load | Skip — llama.cpp will fail with OOM and the error is in journal logs |
**Binary detection fallback:** llama.cpp builds name the binary differently:
```bash
find_llamacpp() {
local candidates=("llama-server" "llama.cpp/server" "server" "llama-server-cuda")
for bin in "${candidates[@]}"; do
command -v "$bin" &>/dev/null && { echo "$bin"; return 0; }
done
return 1
}
```
[DECIDED]
---
## Decision 10: File List & Responsibilities
| File | Action | Responsibility |
|------|--------|----------------|
| `bin/pos-ai-server` | **NEW** | Service manager: start/stop/status/models/logs, systemd unit generation, GPU detection, model selection |
| `lib/ai-providers/llamacpp.sh` | **NEW** | Provider adapter: provider_name, provider_default_model, provider_generate, provider_models_list |
| `bin/pos-ai` | **MODIFY** | Add `llamacpp` cases to resolve_key, resolve_model, cmd_providers, require_key |
| `config/ai.env` | **MODIFY** | Add commented LLAMACPP_* key documentation |
| `completions/pos.bash` | **AUTO** | `make gen` picks up new POS headers — no manual edit |
**NOT in scope:**
- No static `systemd/pos-ai-server.service` file (generated at runtime)
- No changes to `postinstall.sh` (service is user-managed, not installed by system)
- No changes to `bin/pos` dispatcher (tool is auto-discovered)
- No changes to `lib/common.sh`
---
## Decision 11: Implementation Constraints
1. **All `LLAMACPP_*` config reads must go through `load_config()`** — the existing config loader in `pos-ai` (line 131). The new tool also needs its own config loader (or sources `pos-ai`'s, which it can't cleanly). **Decision:** `pos-ai-server` uses its own `load_config()` copy (same pattern as `pos-ai-hf` line 26 — every tool that reads `ai.env` has its own loader).
2. **The systemd unit must NOT hardcode HOME.** The existing pattern (`pos-communication-telegram-listener` line 494-496) explains why: "Do NOT pin Environment=HOME here — the systemd user manager already sets the correct HOME."
3. **ExecStart must use full paths for llama-server** — systemd user services don't inherit the user's full `$PATH`. Resolve via `$(command -v llama-server)` at unit generation time.
4. **The service must use `--log-format` flag** if available (llama.cpp) to produce parseable logs. Not a hard requirement.
5. **`make gen` must run after creating `bin/pos-ai-server`** to regenerate the tree, dispatch table, completions, and doc tables.
6. **The tool must pass `make check && make lint`** — bash -n syntax, exec bit, POS header, --help, deps guards before help.
---
## Verification Plan
1. **Unit test (stub PATH):**
- Fake `llama-server`, `nvidia-smi`, `curl`, `jq` in PATH
- Assert `cmd_start` generates correct unit file content
- Assert `cmd_models` finds `.gguf` files
- Assert `detect_gpu` logic
- Assert config resolution precedence (flag > env > default)
2. **Integration test (manual):**
- `pos ai server start model.gguf` with a real llama-server binary
- `pos ai server status` shows correct info
- `pos ai server logs` shows journal output
- `pos ai server stop` cleans up
- `pos ai --provider llamacpp ask "hello"` routes through local server
3. **Gates:**
- `make check` — green
- `make lint` — 0 FAIL, 0 WARN
---
## Explicitly Out of Scope
- Model conversion/quantization
- Multi-GPU support
- Authentication on the API endpoint
- Web UI
- GPU driver installation
- ROCm/AMD detection
- Quantization awareness (context size vs model capability)
- Model memory estimation / pre-flight checks
- Automatic model download on `start` if none present