diff --git a/.gitignore b/.gitignore index fbba2ae..3ce762e 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,6 @@ session # Scratch/plan notes — never committed TEMP_PLAN.md dream.md + +# Agent process reports — local-only shared memory (not tracked) +AgentsReport/ diff --git a/.n b/.n deleted file mode 100644 index d00491f..0000000 --- a/.n +++ /dev/null @@ -1 +0,0 @@ -1 diff --git a/AGENT_TODO.md b/AGENT_TODO.md index a158bca..d2f44a4 100644 --- a/AGENT_TODO.md +++ b/AGENT_TODO.md @@ -42,6 +42,8 @@ summary (newest last). ## Done +- **2026-09-06** — Repo cleanup: removed 114 temp/process files (AgentsReport/ 86 + reportAgents/ 28 agent reports), stale task/plan/audit docs (tmp_request.md, FINAL_SUMMARY.md, IMPLEMENTATION_PLAN.md, AUDIT.md, AUDIT_TABLE.md, registry-design doc), stray `To`/`.n`/`reports/` artifacts; gitignored AgentsReport/ so agent process reports stay local-only. DOC/, tools-docs/ytsync.md, AGENT_TODO.md, bin/lib/apps/tests/config/scripts etc. kept untouched. + - **2026-09-06** — `pos ai` OpenRouter 402 + unbounded session — Architect→Builder→Reviewer. User hit OpenRouter 402 on the `assist` alias: "You requested up to 131072 tokens, but can only afford 4511" — no provider sent `max_tokens`, so OpenRouter pre-bills the routed model's full worst-case output (131072 on `openrouter/auto`); user also asked to bound session history to last-5 requests. Architect decisions: `AI_MAX_TOKENS` (num, default **2048**, real cost cap) sent as `max_tokens` on OpenRouter and `generationConfig.maxOutputTokens` on Gemini (llamacpp skipped — local/free); `AI_SESSION_TURNS` (messages, 2 per exchange; default 40 kept back-compat; 10 = last 5 conversations) resolved lazily in `session_push` because config loads after the hardcoded line-25 default; both registered in the `bin/pos-ai` `# POS_CONFIG:` `@General` section → visible in `pos config ai`. Reviewer hardening (CHANGES_REQUIRED → fixed): unguarded env input could reach jq `tonumber` (0/-5/010/abc) — both providers + `session_push` now guard with `^[1-9][0-9]*$` fallback-to-default. Verified: fake-curl shim smoke (16 provider-body + 12 session-window checks incl. 010-regression proof), `make gen` idempotent, `make check` OK, `make lint` 0/0, `make test` 17 files/299 checks green; Reviewer ACCEPT (twice). Tester regression round not run this cycle (user's call); permanent coverage remains a follow-up. - **2026-09-06** — `pos ai alias` create-flow silent abort + bogus step labels. User hit: pressing Enter on "System prompt (empty = use built-in)" silently returned to the menu (no alias created); step counters showed `[1/4] [2/4]` in a 5-step flow. Detective: `menu_ask_value` (lib/menu-lib.sh) contract returns rc 1 for empty+no-default, collapsing "empty" with "cancel"; the Step 4 call at `:410` passed `""` default so the advertised empty answer triggered `|| return 0` → silent abort; same latent trap at alias-name `:353` (re-prompt dead code). Pre-existing (introduced with the alias feature `f61766b0`/`9f289ba3`/`300b742a`), NOT a 2026-09-06 regression; 11 other `menu_ask_value` callers correct (6 external rely on empty=cancel, 4 pass defaults) → no global semantic change allowed. Architect: opt-in `--allow-empty` flag on `menu_ask_value` (backward-compatible; empty+no-default → rc 0 empty value, genuine cancel/EOF → rc 1, default wins) + step labels fixed to `/5`. Builder: implemented (lib/menu-lib.sh flag+docs, bin/pos-ai-alias `:353`/`:410` + step counters), 7-case smoke matrix PASS, gen idempotent, check OK, lint 0 FAIL/0 WARN. User chose to commit without the Tester regression round (report: `AgentsReport/tester/2026-09-06_alias-menu-tests.md` — pty feasibility proven, steps 2-6 pending). diff --git a/AUDIT.md b/AUDIT.md deleted file mode 100644 index ddd58e0..0000000 --- a/AUDIT.md +++ /dev/null @@ -1,112 +0,0 @@ -# Audit of POS AI Tools Implementation - -## Current Status Analysis - -Based on my examination of the codebase, I can see that the `pos ai hf` and `pos ai server` tools are implemented but appear to be incomplete compared to the underlying applications they're supposed to wrap. - -## Audit Findings - -### 1. `pos ai hf` - Current State vs. Hugging Face CLI - -**Current Implementation:** -- Supports search, download, list, remove commands -- Basic GGUF filtering capabilities -- Branch/revision support -- Authentication via HF_TOKEN -- Basic file listing and downloading - -**Missing Hugging Face CLI Features:** -- **Repository Discovery:** The tool lacks advanced repository discovery features -- **Model Information:** No `info` or `show` commands to get repository details -- **Advanced Filtering:** Missing `--include` and `--exclude` patterns -- **Revision Support:** Limited branch support, no tag/commit support -- **Cache Management:** No cache inspection or management -- **Authentication:** Only basic token support, no credential helper integration -- **Multi-Shard Models:** Limited support for sharded GGUF models -- **Progress Indicators:** Basic progress, no detailed download metrics -- **Environment Variables:** Limited environment variable handling - -### 2. `pos ai server` - Current State vs. llama.cpp - -**Current Implementation:** -- Supports start, stop, status, models, logs commands -- Basic GPU detection and auto-config -- Port and host configuration -- Model selection -- Basic health checking - -**Missing llama.cpp Features:** -- **GPU Configuration:** No support for detailed GPU offloading (`-ngl`, multi-GPU) -- **Memory Management:** No support for KV cache configuration -- **Performance Options:** Missing batch sizes, ubatch sizes, continuous batching -- **Sampling Parameters:** No temperature, top-k, top-p, repetition penalty controls -- **Advanced Features:** Missing JSON/schema, tool calling, reasoning options -- **Server Configuration:** No support for all server options like metrics, health endpoints -- **Version Detection:** No capability to detect and validate llama-server version -- **Graceful Shutdown:** Limited shutdown handling - -## Technical Limitations - -### Hugging Face CLI Analysis -Based on the Hugging Face documentation and typical CLI patterns, the actual `hf` command provides: -- `hf list` - List repositories -- `hf info ` - Show repository information -- `hf files ` - List repository files -- `hf download ` - Download repository with various options -- `--include`/`--exclude` - File pattern filtering -- `--revision` - Specific revision support -- `--cache` - Cache management -- Authentication with tokens or credential helpers -- Detailed progress reporting - -### llama.cpp Analysis -Based on llama.cpp documentation, the actual `llama-server` provides: -- `--model ` - Model file specification -- `--port ` - Port binding -- `--host ` - Host binding -- `--ctx-size ` - Context size -- `--n-gpu-layers ` - GPU layer count -- `--tensor-split` - Tensor split configuration -- `--split-mode` - Split mode (none, layer, row) -- `--flash-attn` - Flash attention support -- `--threads ` - Thread count -- `--mmap` - Memory mapping -- `--mlock` - Memory locking -- `--batch-size` - Batch size -- `--ubatch-size` - UBatch size -- `--log-disable` - Logging control -- `--health` - Health endpoint -- `--slots` - Concurrent request slots -- `--metrics` - Metrics endpoint - -## Required Improvements - -### For `pos ai hf`: - -1. **Add Info Command**: `pos ai hf info ` -2. **Add Files Command**: `pos ai hf files ` -3. **Add Cache Management**: `pos ai hf cache` -4. **Enhance Download**: Support include/exclude, revision, and better progress -5. **Model Information**: Show model details, size, and metadata -6. **Repository Files**: List files with size and metadata -7. **Authentication**: Better credential handling -8. **Version Support**: Detect and support version-specific features - -### For `pos ai server`: - -1. **GPU Configuration**: Support detailed GPU offloading parameters -2. **Memory Management**: Context size, KV cache, memory allocation -3. **Performance Tuning**: Batch size, ubatch size, continuous batching -4. **Sampling Controls**: Temperature, top-k, top-p, repetition penalty -5. **Advanced Features**: JSON/schema, tool calling, reasoning -6. **Server Options**: Health, metrics, concurrency control -7. **Version Awareness**: Detect and validate supported options -8. **Process Management**: Better monitoring and graceful shutdown - -## Implementation Approach - -Given that we don't have the actual underlying CLI tools installed in this environment, I'll need to: -1. Create a comprehensive audit document -2. Design the proper interface based on documented capabilities -3. Implement stubs and placeholders for actual functionality -4. Ensure all the missing features are properly accounted for in the plan \ No newline at end of file diff --git a/AUDIT_TABLE.md b/AUDIT_TABLE.md deleted file mode 100644 index 7810280..0000000 --- a/AUDIT_TABLE.md +++ /dev/null @@ -1,166 +0,0 @@ -# Detailed Audit of POS AI Tools vs. Underlying Applications - -## 1. `pos ai hf` - Audit Table - -| Category | Current Implementation | Upstream Hugging Face CLI | Missing | Incorrect | Fix | -|----------|----------------------|---------------------------|---------|-----------|-----| -| **Repository Discovery** | Basic search | `hf list`, `hf info` | `info` command | - | Add `pos ai hf info ` | -| **File Listing** | `--list` flag | `hf files` | `files` command | - | Add `pos ai hf files ` | -| **Model Information** | Basic metadata | Detailed model info | No detailed info | - | Add model details display | -| **Repository Files** | Limited listing | `hf files` with sizes | No file listing | - | Add file listing command | -| **Downloading** | Basic download | `hf download` with include/exclude | No pattern filtering | - | Add `--include`/`--exclude` | -| **Revisions** | `--branch` | `--revision` support | No tag/commit support | - | Add revision support | -| **Cache Management** | No cache commands | `hf cache` | No cache commands | - | Add `pos ai hf cache` | -| **Authentication** | `HF_TOKEN` only | Credential helpers, more tokens | Limited auth | - | Add enhanced auth | -| **Progress Reporting** | Basic progress | Detailed metrics | Limited info | - | Add progress details | -| **File Pattern Matching** | Basic file matching | Wildcards, patterns | No pattern support | - | Add pattern matching | -| **Multi-Shard Support** | Limited | Full sharded support | No sharded model support | - | Add sharded model support | -| **Error Handling** | Basic errors | Detailed error messages | Limited context | - | Improve error reporting | - -## 2. `pos ai server` - Audit Table - -| Category | Current Implementation | Upstream llama.cpp | Missing | Incorrect | Fix | -|----------|----------------------|--------------------|---------|-----------|-----| -| **Model Loading** | Basic model selection | `--model` with validation | - | - | Enhance model validation | -| **GPU Offloading** | Auto-detect | `--n-gpu-layers`, multi-GPU | No detailed GPU control | - | Add GPU layer control | -| **Memory Context** | Basic context | `--ctx-size`, KV cache | Limited memory options | - | Add memory controls | -| **Performance Tuning** | Basic params | Batch sizes, threads, ubatch | No tuning | - | Add performance options | -| **Sampling Controls** | Basic | Temperature, top-k, top-p | No sampling options | - | Add sampling parameters | -| **Advanced Features** | No advanced | JSON/schema, tool calling | No advanced features | - | Add advanced options | -| **Server Configuration** | Basic | Health, metrics, concurrency | Limited server options | - | Add server options | -| **Version Detection** | No version | `--version` support | No version awareness | - | Add version detection | -| **Process Management** | Basic | Graceful shutdown, monitoring | Limited process control | - | Add process monitoring | -| **Logging** | Basic | Logging controls | No log options | - | Add logging controls | - -## 3. Detailed Missing Features - -### For `pos ai hf`: - -**Missing Repository Information:** -- No `info` command to show repository details -- No way to see model architecture or parameters -- No commit history or version details - -**Missing File Operations:** -- No `files` command to list repository contents -- No file pattern matching or filtering -- No file size information in listings - -**Missing Advanced Downloading:** -- No `--include`/`--exclude` patterns -- No revision/tag support -- No cache management commands - -**Missing Authentication:** -- No credential helper support -- No token validation -- No multi-auth method support - -### For `pos ai server`: - -**Missing GPU Configuration:** -- No `--n-gpu-layers` support -- No tensor splitting (`--tensor-split`) -- No multi-GPU configuration -- No GPU device selection - -**Missing Memory Management:** -- No `--ctx-size` control -- No KV cache configuration -- No memory mapping (`--mmap`) or locking (`--mlock`) - -**Missing Performance Options:** -- No batch size control (`--batch-size`) -- No ubatch size (`--ubatch-size`) -- No continuous batching options - -**Missing Sampling Controls:** -- No temperature control -- No top-k, top-p options -- No repetition penalty -- No JSON/schema support - -**Missing Server Features:** -- No health endpoint configuration -- No metrics endpoint support -- No concurrency controls -- No graceful shutdown handling - -**Missing Version Awareness:** -- No version detection capability -- No feature compatibility checking -- No version-specific option support - -## 4. Implementation Priority - -### Critical (Must Have): -1. Add `info` and `files` commands for `pos ai hf` -2. Add GPU control for `pos ai server` -3. Add version detection -4. Add proper error handling - -### High Priority: -1. Add `--include`/`--exclude` patterns -2. Add revision support -3. Add memory context control -4. Add performance tuning options - -### Medium Priority: -1. Add cache management -2. Add enhanced authentication -3. Add advanced sampling -4. Add advanced server configuration - -### Low Priority: -1. Add progress metrics -2. Add logging controls -3. Add process monitoring -4. Add integration with POS ecosystem - -## 5. Technical Requirements - -### For `pos ai hf`: -- Enhanced parsing for new flags -- Integration with Hugging Face API -- Cache management system -- Better file pattern matching -- Improved error reporting - -### For `pos ai server`: -- Enhanced parameter parsing -- Version detection system -- GPU configuration module -- Memory management controls -- Process management system -- Configuration validation - -## 6. Compatibility Considerations - -### Backward Compatibility: -- All existing commands must work unchanged -- All existing options must work unchanged -- Default behavior must be preserved -- Configuration file compatibility maintained - -### Breaking Changes: -- None planned -- All enhancements are additive -- No existing functionality removed - -## 7. Testing Requirements - -### `pos ai hf` Testing: -- Model download functionality -- File pattern matching -- Revision handling -- Cache operations -- Authentication testing - -### `pos ai server` Testing: -- GPU configuration validation -- Memory parameter testing -- Performance tuning options -- Server startup/shutdown -- Version compatibility testing - -This audit identifies the comprehensive gap between the current POS tools and the capabilities of the underlying Hugging Face CLI and llama.cpp server. The implementation plan will address all these gaps systematically. \ No newline at end of file diff --git a/AgentsReport/architect/2026-08-26_ai-alias-architecture.md b/AgentsReport/architect/2026-08-26_ai-alias-architecture.md deleted file mode 100644 index e5b2ae2..0000000 --- a/AgentsReport/architect/2026-08-26_ai-alias-architecture.md +++ /dev/null @@ -1,527 +0,0 @@ -# Architecture: `pos ai alias` — AI Agent Alias Manager - -**Date:** 2026-08-26 -**Author:** Architect (big-pickle) -**Status:** DECISION_READY - ---- - -## TL;DR - -| Decision | Choice | -|----------|--------| -| Storage | `~/.config/linux_post_install/ai-aliases.env` (pipe-delimited structured data) | -| Shell aliases | `~/.config/linux_post_install/ai-aliases.sh` (generated, never hand-edited) | -| Tool | `bin/pos-ai-alias` — new `ai` category tool | -| No POS_CONFIG scope | Alias management is CRUD, not env-key editing; `pos config` is not involved | -| Source of truth | `.env` file; `.sh` file is regenerated on every write | -| .bashrc integration | One conditional `source` line, added by postinstall.sh | -| INSTALL_CHANGES | `postinstall.sh` adds `.bashrc` source line; `bin/pos` adds `ai-alias` to INTERACTIVE_CMDS | - -**Open items:** -- None — all decisions are evidence-backed from project conventions. - ---- - -## Decision 1: Data Format - -**Problem:** Store named AI aliases (provider, session name, system prompt) durably, with safe special-character handling and trivial parsing. - -### Options Considered - -**Option A: Shell alias format (one-liner shell aliases)** -``` -alias devbot='pos ai gemini ask --session devbot --system "You are a dev assistant"' -``` -- Architecture: Store raw shell alias lines; `.bashrc` sources the file directly. -- Advantages: No generation step; shell sources it natively. -- Costs: Parsing aliases back into components (for edit/list) requires fragile quote-aware shell parsing; single quotes inside system prompts break the syntax. -- Risks: Prompt containing `'` corrupts the file. Edit must read then reconstruct — fragile. - -**Option B: Pipe-delimited structured data + generated .sh** -``` -devbot|gemini|devbot|You are a dev assistant -``` -- Architecture: `.env` file is the source of truth (pipe-delimited fields). A separate `.sh` file is regenerated from it on every write. `.bashrc` sources the `.sh` file. -- Advantages: Parsing is trivial (`IFS='|'`); single-quote escaping is handled at generation time; data is safe to `grep`/`sort`/`awk`. -- Costs: One extra file (`.sh`); a `_regen_aliases()` helper function. -- Risks: Generation must escape correctly — but this is a single, testable function. - -**Option C: Individual env files per alias** -``` -~/.config/linux_post_install/ai-aliases/devbot.env -``` -- Architecture: One file per alias; load all at shell startup. -- Advantages: No parsing of multi-entry files. -- Costs: Directory management; glob at shell startup; harder to list all; no atomic operations. - -### Decision: Option B - -Pipe-delimited structured data is the cleanest separation. The `.env` file is the source of truth. The `.sh` file is a generated artifact. This matches the project's own pattern of "generated code between GEN markers" — except here the generator lives inside the tool itself, not `make gen`. - -**Rationale:** The project already has a pattern of generated files (e.g., `GEN:START`/`GEN:END` blocks, `completions/pos.bash`). The tool owns its own generation. Parsing shell aliases is fragile and error-prone; pipe-delimited data is trivially safe. - -### Data Format Specification - -``` -# ~/.config/linux_post_install/ai-aliases.env -# Managed by: pos ai alias (do not hand-edit) -# Format: alias_name|provider|session_name|system_prompt -# Pipe characters in system_prompt are not supported. -# -# agent_name|provider|session_name|system_prompt -devbot|gemini|devbot|You are a Linux dev assistant. Reply with commands only. -code|openrouter|codereview|You are a code reviewer. Be concise. -``` - -**Field constraints:** - -| Field | Rules | -|-------|-------| -| `alias_name` | Shell-valid identifier: `[a-zA-Z][a-zA-Z0-9_-]*` | -| `provider` | Must match an installed provider: `gemini`, `openrouter`, etc. | -| `session_name` | Defaults to the alias name if empty; `[a-zA-Z0-9_-]+` | -| `system_prompt` | Free text; no `\|` (pipe) characters; may be empty (uses built-in prompt) | - -**Header:** Two comment lines at the top (file description + format) are auto-maintained by the tool. - -**Empty session_name convention:** When session_name is empty (field is blank between pipes), the alias uses the alias name as the session name. This avoids redundant repetition for the common case. - ---- - -## Decision 2: File Layout - -### New files - -| File | Purpose | -|------|---------| -| `bin/pos-ai-alias` | New CLI tool (100755) | - -### Modified files - -| File | Change | -|------|--------| -| `bin/pos` | Add `ai-alias` to `INTERACTIVE_CMDS` list | -| `postinstall.sh` | Add `.bashrc` source line for `ai-aliases.sh` (conditional, no-clobber) | - -### Runtime files (user config, NOT in repo) - -| File | Purpose | Mutability | -|------|---------|------------| -| `~/.config/linux_post_install/ai-aliases.env` | Alias data (source of truth) | Created/modified by tool | -| `~/.config/linux_post_install/ai-aliases.sh` | Generated shell aliases | Regenerated on every write | - -### Files NOT modified - -| File | Why not | -|------|---------| -| `.gitignore` | `~/.config/linux_post_install/` is a user directory, not in the repo. No new repo files to ignore. | -| `config/` | No template file needed — the env file is user-created on first use. | -| `lib/` | No new library. Tool sources `common.sh` + `menu-lib.sh` from existing libs. | - ---- - -## Decision 3: Tool Interface - -### POS Header - -```bash -# POS: ai alias — Create/edit/remove named AI agent aliases -# POS_SUBCMDS: create edit remove list show -``` - -No `POS_FLAGS:` — this is a subcommand-based tool, not a flag-based tool. - -### CLI Interface - -``` -pos ai alias [subcommand] [args] - -Subcommands: - (no args) Interactive menu (create/edit/remove/list) - create Create a new alias (interactive prompts) - create Create with given name (interactive prompts for rest) - edit Pick an alias to edit (interactive) - edit Edit a specific alias - remove Pick an alias to remove (interactive, with confirmation) - remove Remove a specific alias (with confirmation) - list List all aliases (non-interactive, machine-readable) - show Show one alias's details - -Options: - -h|--help Show this help. - -Examples: - pos ai alias # interactive menu - pos ai alias list # show all aliases - pos ai alias create # interactive create - pos ai alias create mybot # create 'mybot' alias - pos ai alias edit mybot # edit the 'mybot' alias - pos ai alias remove mybot # remove 'mybot' (with confirm) - pos ai alias show mybot # show alias details -``` - -### Subcommand Details - -**`pos ai alias` (no args):** Interactive menu using `menu_run` from `lib/menu-lib.sh`. Options: -1. Create new alias -2. Edit existing alias -3. Remove alias -4. List aliases - -**`pos ai alias list`:** Non-interactive table output: -``` -Aliases (3): - mybot gemini mybot You are a helpful assistant - code openrouter code You are a code reviewer - dev gemini devbot Dev assistant -``` -Format: `%-12s %-12s %-12s %s` (name, provider, session, prompt-truncated-to-60). - -**`pos ai alias show `:** Full details including the resolved shell command. - ---- - -## Decision 4: POS_CONFIG - -**Decision: NO POS_CONFIG scope.** - -**Rationale:** The `pos config` / `POS_CONFIG` system is designed for simple `KEY=VALUE` env files (like `ai.env`, `telegram.env`, `entertainment.env`). Aliases are structured, multi-field records, not key-value pairs. The `cfg_ui()` pattern from `lib/config-ui.sh` doesn't apply here — it renders a numbered menu of KEY=VALUE pairs, not CRUD operations on named records. - -The alias tool is self-contained with its own interactive menus. It does not participate in `pos config`. - ---- - -## Decision 5: Shell Alias Generation - -### The `_regen_aliases()` function - -This function reads `ai-aliases.env` and writes `ai-aliases.sh`: - -```bash -_regen_aliases() { - local env_file="$1" sh_file="$2" tmp - tmp="$(mktemp)" - printf '#!/usr/bin/env bash\n# Auto-generated by pos ai alias — do not hand-edit.\n# Source: %s\n\n' "$env_file" >"$tmp" - - if [ -f "$env_file" ]; then - while IFS='|' read -r name provider session prompt _rest; do - # Skip comments and empty lines - [[ "$name" =~ ^[[:space:]]*# ]] && continue - [[ -z "$name" ]] && continue - - # Validate alias name - [[ "$name" =~ ^[a-zA-Z][a-zA-Z0-9_-]*$ ]] || continue - - # Session defaults to alias name if empty - [ -z "$session" ] && session="$name" - - # Escape single quotes in the system prompt for shell-safe embedding - local escaped_prompt="${prompt//\'/\'\\\'\'}" - - printf "alias %s='pos ai %s ask --session %s" "$name" "$provider" "$session" >>"$tmp" - if [ -n "$prompt" ]; then - printf " --system '%s'" "$escaped_prompt" >>"$tmp" - fi - printf "'\n" >>"$tmp" - done < <(grep -v '^[[:space:]]*#' "$env_file" | grep -v '^[[:space:]]*$' || true) - fi - - mv "$tmp" "$sh_file" - chmod 644 "$sh_file" -} -``` - -**Single-quote escaping:** `${prompt//\'/\'\\\'\'}` — bash `parameter expansion` replaces every `'` with `'\''` (close-quote, escaped-quote, open-quote). This is the standard and safe pattern for embedding arbitrary strings in single-quoted shell contexts. - -**Edge case — empty prompt:** When the system prompt is empty, the `--system` flag is omitted entirely, letting `pos ai` use its built-in default prompt. - -### Syntax validation before commit - -After generating the `.sh` file, run `bash -n` to verify syntax: -```bash -if ! bash -n "$sh_file" 2>/dev/null; then - warn "Generated alias file has syntax errors — keeping previous version" - rm -f "$tmp" - return 1 -fi -``` - -### Shell integration in .bashrc - -```bash -# AI aliases (managed by pos ai alias) -[ -f ~/.config/linux_post_install/ai-aliases.sh ] && source ~/.config/linux_post_install/ai-aliases.sh -``` - -This line is added by `postinstall.sh` with the standard no-clobber grep check. - ---- - -## Decision 6: Interactive Flow - -### Main Menu (`pos ai alias` — no args) - -Uses `menu_run` from `lib/menu-lib.sh`: - -``` -════════════════════════════════════════════ - AI Agent Aliases -════════════════════════════════════════════ - 1) Create new alias - 2) Edit existing alias - 3) Remove alias - 4) List aliases - 0) Exit ----------------------------------------- -Choose: -``` - -### Create Flow - -1. Prompt for alias name: `menu_ask_value "Alias name" ""` — validate format (`[a-zA-Z][a-zA-Z0-9_-]*`) -2. Check for duplicate name → warn and re-prompt if taken -3. Prompt for provider: show available providers (read from `$PROVIDER_DIR/*.sh`), default `gemini` -4. Prompt for session name: default = alias name -5. Prompt for system prompt: default = empty (uses built-in) -6. Confirm: `confirm "Create alias ''?" y` -7. Write to `.env`, regenerate `.sh`, log success - -### Edit Flow - -1. List existing aliases (name + provider + first-40-chars of prompt) -2. Pick one (if no arg given): `menu_pick "Pick alias" "${names[@]}"` -3. Show current values -4. For each field, prompt with current value as default (Enter = keep) -5. Confirm changes -6. Rewrite `.env` entry, regenerate `.sh` - -### Remove Flow - -1. Pick alias: `menu_pick` or named -2. Show alias details -3. `confirm "Remove alias ''? This cannot be undone." n` (default = no) -4. Remove from `.env`, regenerate `.sh` - -### List Flow (non-interactive) - -Prints formatted table to stdout. Provider column left-aligned, name left-aligned, prompt truncated to 60 chars with `...`. - -### Show Flow (non-interactive) - -Full details + resolved command: -``` -Alias: mybot -Provider: gemini -Session: mybot -Prompt: You are a helpful assistant -Command: pos ai gemini ask --session mybot --system 'You are a helpful assistant' -``` - ---- - -## Decision 7: Edge Cases and Error Handling - -| Edge Case | Handling | -|-----------|----------| -| **Duplicate alias name on create** | `warn "Alias '$name' already exists — use 'pos ai alias edit $name' instead"`; re-prompt | -| **Empty alias name** | `err "Alias name cannot be empty"` | -| **Invalid alias name** (contains spaces, starts with digit) | `err "Invalid alias name '$name' — use letters, digits, hyphens, underscores"` | -| **Invalid provider** | `err "Unknown provider '$p' — available: $(ls ...)"` — uses same provider discovery as `pos-ai` | -| **Empty system prompt** | Allowed — omit `--system` flag; `pos ai` uses its built-in default prompt | -| **Very long system prompt** | Truncate display in `list` output (60 chars + `...`); full value preserved in `.env` and `.sh`. Warn if > 500 chars during creation. | -| **Pipe character in system prompt** | Rejected on input: `err "System prompt must not contain '\|' characters"` | -| **System prompt with single quotes** | Handled by `_regen_aliases()` escaping: `'` -> `'\''` in the generated alias line | -| **Editing an alias "in use"** | No lock/detection needed. The user edits the `.env`; on next shell startup (or `source ~/.bashrc`), aliases update. No runtime state conflict. | -| **File doesn't exist yet** | First create auto-creates both `.env` and `.sh` | -| **Corrupted/invalid .env line** | Skipped by `_regen_aliases()` (name validation regex) | -| **Concurrent edits** | Not a concern — personal single-user tool. Last write wins. | -| **Missing pos-ai dependency** | Tool doesn't require `pos-ai` at runtime — it only writes config. No dep guard needed. | - ---- - -## Decision 8: Security Considerations - -### System prompt injection - -System prompts are user-authored text that becomes a shell argument. Risks: -- **Shell injection via alias execution:** The prompt is single-quoted in the alias, so shell metacharacters (`$`, backtick, `!`) are literal — safe. -- **`pos ai` prompt injection:** This is a user's own prompt for their own AI. No trust boundary crossing. -- **File permissions:** `ai-aliases.env` gets `chmod 600` (user-only read; consistent with other config files). `ai-aliases.sh` gets `chmod 644` (needed by bash `source`). - -### Escaping correctness - -The single-quote escaping `${prompt//\'/\'\\\'\'}` is the only place where correctness matters critically. If it fails, the generated alias has a syntax error and `source` will report it. Mitigation: -- The tool runs `bash -n` on the generated `.sh` file before committing it. -- If syntax check fails, warn and skip the regeneration (keep the old `.sh`). - -### No secrets in the alias file - -System prompts are not secrets — they're user-authored instructions. API keys stay in `ai.env` (already managed by `pos config ai`). No new secret surface. - ---- - -## Decision 9: Integration with pos-ai - -### How aliases invoke pos-ai - -Each generated alias calls: -```bash -alias ='pos ai ask --session --system ""' -``` - -This uses the existing `pos-ai` flags: -- `--provider ` — supported since the beginning (line 633 of `bin/pos-ai`) -- `--session ` — supported (line 639) -- `--system ` — supported (line 642) - -**No changes to `bin/pos-ai` are required.** The alias tool is a standalone configuration tool that writes shell aliases calling `pos ai`. - -### Provider validation - -The alias tool must validate the provider name against installed providers. It reuses the same discovery logic from `pos-ai`: -```bash -PROVIDER_DIR="$(dirname "$0")/../lib/ai-providers" -# Fallback for installed layout -[ -d "$PROVIDER_DIR" ] || PROVIDER_DIR="$(dirname "$0")/ai-providers" -``` - -This is the same pattern used in `pos-ai` at line 17-18. The alias tool discovers providers independently (no dependency on `pos-ai` being sourced). - ---- - -## Decision 10: Installation Changes - -### postinstall.sh modification - -Add a block after the existing `ai.env` installation (around line 50): - -```bash -# ── AI aliases shell integration ─────────────────────────────── -ALIAS_SRC_LINE='# AI aliases (managed by pos ai alias) -[ -f ~/.config/linux_post_install/ai-aliases.sh ] && source ~/.config/linux_post_install/ai-aliases.sh' -if ! grep -qsF "ai-aliases.sh" "$BASHRC" 2>/dev/null; then - run printf '%s\n' "$ALIAS_SRC_LINE" >> "$BASHRC" - log "Added AI aliases source to ~/.bashrc" -fi -``` - -### bin/pos modification - -Add `ai-alias` to the `INTERACTIVE_CMDS` list (line 262). Current value: -``` -INTERACTIVE_CMDS="docker-compose docker-vbox network-hotspot system-firewall media-mp4 media-sync system-backup system-uninstall share-usb-server share-smb-server share-smb-client share-nfs-client share-nfs-server communication-telegram-listener communication-matrix-listener ai ai-gemini ai-openrouter system-schedule entertainment-config config" -``` - -Add `ai-alias` to this space-separated list. - -### No other installation changes - -- No new apt packages (no deps beyond bash) -- No new lib files (tool sources `common.sh` + `menu-lib.sh` from existing libs) -- No systemd services -- No config template in `config/` - ---- - -## Decision 11: Long Prompt Handling - -System prompts can be arbitrarily long. Shell aliases have a practical limit (ARG_MAX, typically 2MB on Linux), so this is not a hard constraint. However: - -- **Display:** `list` output truncates to 60 chars + `...` -- **Storage:** Full prompt in `.env` and `.sh` — no truncation -- **Interactive edit:** Shows full current value, allows full editing -- **Warning:** During creation, if prompt exceeds 500 chars: `warn "System prompt is long (${#prompt} chars) — consider keeping it concise"` - -**Decision:** No artificial length limit. - ---- - -## Implementation Phases - -### Phase 1: Core tool (single commit) - -1. Create `bin/pos-ai-alias` with: - - Shebang, strict mode, `common.sh` source, `menu-lib.sh` source - - `# POS:` header + `# POS_SUBCMDS:` - - `CONFIG_FILE` and `ALIASES_SH_FILE` path constants - - `_load_aliases()` — reads `.env` into parallel arrays (names, providers, sessions, prompts) - - `_find_alias()` — lookup by name, returns index - - `_write_env_file()` — writes entire `.env` from arrays - - `_regen_aliases()` — reads `.env`, writes `.sh` with proper escaping - - `_list_aliases()` — non-interactive table output - - `_show_alias()` — non-interactive single alias details - - `_create_alias()` — interactive create with validation - - `_edit_alias()` — interactive edit with field-level prompts - - `_remove_alias()` — interactive remove with confirm - - `_main_menu()` — interactive menu via `menu_run` - - Subcommand dispatch (`case` pattern) - - `usage()` function -2. Add `ai-alias` to `INTERACTIVE_CMDS` in `bin/pos` -3. Add `.bashrc` source line to `postinstall.sh` - -### Phase 2: Verification - -1. `chmod +x bin/pos-ai-alias` -2. `bash -n bin/pos-ai-alias` -3. `make gen` — regenerate tables (new tool appears in dispatch table, bin tree, file table) -4. `make check` — self-consistency gate -5. `make lint` — convention gate (0 FAIL, 0 WARN) -6. Manual test: create, list, show, edit, remove aliases; verify `.sh` file is correct -7. Source `.bashrc` and verify aliases work - -### Phase 3: Documentation - -1. `DOC/POS.md` — add `ai alias` section (hand-written) -2. `DOC/HOWTO.md` — add index row -3. `DOC/howto/ai.md` — add aliases section (if ai.md exists; otherwise add to existing ai howto) -4. `DOC/AGENT_Context_Project.md` — regenerated by `make gen`; hand-add to Common Tasks table -5. Update `AGENT_TODO.md` Done section (dated) - ---- - -## Architectural Constraints - -1. **Tool must source `lib/common.sh`** via the fallback chain (not self-contained) -2. **Tool must source `lib/menu-lib.sh`** for interactive menus -3. **Tool MUST be in `INTERACTIVE_CMDS`** in `bin/pos` -4. **Generated `.sh` file must pass `bash -n`** before commit -5. **Env-seam:** all file paths use `${CONFIG_DIR:-...}` pattern (already in `common.sh`) -6. **`chmod 600`** for `.env`, `chmod 644` for `.sh` -7. **No dependency on `pos-ai`** being installed — tool writes config, doesn't run `pos ai` -8. **Pipe delimiter** — system prompts must not contain `|`; validated on input - ---- - -## Verification Checklist - -- [ ] `bash -n bin/pos-ai-alias` passes -- [ ] `shellcheck bin/pos-ai-alias` passes (or only known false positives) -- [ ] `make gen` regenerates tables correctly -- [ ] `make check` passes (bash -n, exec bits, doc sync, smoke) -- [ ] `make lint` passes (0 FAIL, 0 WARN) -- [ ] `pos ai alias --help` shows help -- [ ] `pos ai alias` shows interactive menu -- [ ] Create -> list -> show -> edit -> remove cycle works -- [ ] Generated `.sh` file has correct alias syntax -- [ ] `bash -n` on generated `.sh` passes -- [ ] `.bashrc` source line works (aliases available after source) -- [ ] Special chars in system prompt (single quotes, spaces, $) survive round-trip -- [ ] Duplicate name is rejected -- [ ] Invalid alias name is rejected -- [ ] Invalid provider is rejected - ---- - -## Risks and Mitigations - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| Single-quote escaping fails for exotic prompts | Low | High (syntax error in .sh) | `bash -n` check before commit; warn + skip on failure | -| User has many aliases -> list becomes long | Low | Low | `menu_pick` already supports filtering | -| `.bashrc` source line conflicts with existing alias definitions | Very Low | Medium | Grep-check before adding; line is a conditional source, not an alias definition | -| `make lint` rejects the new tool for a convention violation | Low | Low (blocking) | Follow template exactly; deps guards before help; proper header | - ---- - -**End of architecture document.** diff --git a/AgentsReport/architect/2026-08-26_alias-architecture.md b/AgentsReport/architect/2026-08-26_alias-architecture.md deleted file mode 100644 index 6d85568..0000000 --- a/AgentsReport/architect/2026-08-26_alias-architecture.md +++ /dev/null @@ -1,223 +0,0 @@ -# Architecture: `pos ai alias` — robust activation mechanism - -**Date:** 2026-08-26 -**Author:** Architect (ox-alpha) -**Status:** DECISION_READY (supersedes activation decisions in `AgentsReport/architect/2026-08-26_ai-alias-architecture.md`) - ---- - -## TL;DR - -| Decision | Choice | -|----------|--------| -| D1 — Activation artifact | **Option B**: executable wrapper scripts at `~/.local/bin/`, generated from the ENV source of truth. No shell aliases. | -| D2 — Migration | Internal `_alias_sync()` runs on every `pos ai alias` invocation; reconciles wrappers ↔ ENV both directions; legacy `ai-aliases.sh` auto-removed when marker-guarded, else manual guidance. `.sh` generation stops entirely (no deprecated shim). | -| D3 — Edge cases | Name collisions refused (foreign file or other binary on PATH); empty set → sync deletes all owned wrappers; dead provider → existing `pos-ai:121` error is sufficient; list/show formats unchanged, show gains wrapper path. | -| D4 — Scope fence | Only `bin/pos-ai-alias` + small marker-scan addition to `bin/pos-system-uninstall` + docs. NOT: `bin/pos-ai`, ENV format, config-ui, postinstall.sh, menu flows. | -| Postinstall gap | The deferred `.bashrc` wiring is closed by **obsolescence**, not implementation — no wiring needed anymore. | - -**Open items:** None blocking. One provisional nuance in D2 (legacy-file heuristic guard) flagged inline. - ---- - -## Problem statement - -- Aliases were frozen at source-time: editing provider gemini→openrouter left the stale alias live in running shells → Gemini 429 while `list` showed openrouter (live failure). -- Activation required a `.bashrc` source line that postinstall never wired — the feature was broken-by-omission even before staleness. -- Non-interactive contexts (cron, scripts, ssh non-login) could never use aliases at all. - -## Evidence base - -- `bin/pos-ai-alias:62-104` — current `_alias_regen()` writes `ai-aliases.sh`; success messages at :308 say "Reload shell: source ~/.bashrc". -- `postinstall.sh:80` — PATH export already includes `$HOME/.local/bin` on pos-managed machines (Debian default `~/.profile` also adds it when the dir exists). -- `bin/pos-system-uninstall:96,261` — established precedent for marker-managed user-local binaries (`$HOME/.local/bin/pos-ai-hook.sh`): discovery list + removal pass already exist as a pattern. -- `bin/pos-ai:658-665, :118-121` — provider resolution flag > env > default; unknown provider yields `err "Unknown provider '$p' — available: …"` (good runtime failure quality, no pos-ai change needed). -- Prior architect report D5/D10 chose `.sh` + `.bashrc` wiring; the wiring was never implemented. This report supersedes those two decisions; storage (ENV format) and CRUD UX decisions carry forward unchanged. - ---- - -## Decision 1: Activation artifact - -### Options evaluated - -**Option A: keep generated bash aliases + wire `.bashrc` + louder hints + unalias guards** -- Advantages: smallest diff; familiar alias UX. -- Costs: staleness is *inherent* — the artifact is a snapshot copied into each shell at source time; the live failure (stale gemini alias) can only be mitigated, never eliminated. Requires postinstall `.bashrc` wiring (the deferred gap), reload-hint UX that demonstrably fails ("users miss it"), and per-shell unalias guard logic for a mechanism bash makes awkward to retract. -- Risks: cron/scripts/ssh non-interactive shells get nothing. Two truths (env file vs sourced copy) persist forever. -- Compatibility impact: none. Operational impact: permanent "did you re-source?" support burden. - -**Option B: executable wrapper scripts at `~/.local/bin/`** -- Architecture: ENV stays the single source of truth; tool renders one standalone script per alias: - ```bash - #!/usr/bin/env bash - # Managed by pos ai alias — regenerated automatically; hand-edits are overwritten. - # Alias: assist | provider: openrouter | session: assist - set -euo pipefail - exec pos ai openrouter ask --session assist --system "$@" - ``` - (empty prompt → omit `--system`; prompt embedded via existing double-layer `printf %q` mechanics, reused from `_alias_quote_cmd`, so it lands as exactly ONE shell word; `"$@"` passes user args through). -- Advantages: **staleness eliminated** — next invocation reads current file bytes; no shell integration of any kind (kills the postinstall gap instead of closing it); works identically in interactive shells, scripts, cron, ssh non-login; removal = delete one marker-identified file; no INTERACTIVE_CMDS/tee-pipe interaction changes; content edits bypass `hash` caching entirely (bash caches paths, not contents) and new names are found on first PATH scan. -- Costs: PATH-presence dependency (`~/.local/bin` must be on PATH — already guaranteed by `postinstall.sh:80` and Debian default `~/.profile`, but needs a runtime check + guidance); collision policy needed (scripts are filesystem entries, aliases weren't); ~40 lines more logic than Option A. -- Risks: name shadows a real binary → mitigated by refusal policy (D3); user hand-edits wrapper → healed by sync regeneration (D2), and the header says so. -- Compatibility impact: activation semantics change (documented). Operational impact: self-healing artifacts; zero shell-config coupling. Migration impact: handled by D2. - -**Option C: hybrid — wrappers primary + optional still-generated alias file** -- Advantages: covers users attached to aliases. -- Costs: keeps the stale-snapshot mechanism alive alongside the fix — two activation paths, two truth-drift surfaces, double the validation matrix. Directly contradicts the motivation ("more robust" = fewer failure modes, not one more). -- Risks: the exact reported bug remains reachable through the optional path. - -### Decision: Option B - -Staleness was an architectural property of source-time snapshots, not an implementation bug — no amount of hints or guards fixes Option A. Option B removes the class of bug (artifact always equals source of truth at invocation time) and deletes the deferred `.bashrc` wiring requirement rather than implementing it. Option C preserves the bug class for zero new capability. Smallest robust design wins. - -**Artifact specification (binding for Builder):** - -| Property | Value | -|----------|-------| -| Location | `${HOME}/.local/bin/` | -| Permissions | `0755` | -| Ownership marker | Line 2 contains literal `Managed by pos ai alias` (grep target for all ownership checks) | -| Body | `set -euo pipefail` + single `exec pos ai ask --session [ --system <%q prompt>] "$@"` | -| Quoting | Reuse `_alias_quote_cmd` verbatim (double-layer `%q` mechanics preserved per brief) | -| Atomic write | mktemp in same dir → `mv` → `chmod 755` (same pattern as current `_alias_regen`) | -| Pre-commit validation | `bash -n` on rendered wrapper; on failure warn + keep previous file | -| Secrets | None inside (prompt is content, not credential) | - -[DECIDED] - -## Decision 2: Migration & back-compat - -### Regeneration trigger: `_alias_sync()` on every invocation - -New internal function, called at the top of **every** subcommand dispatch entry (`create`, `edit`, `remove`, `list`, `show`, interactive menu) before the subcommand's own logic. It reconciles `~/.local/bin` against the ENV file in both directions: - -1. For each ENV entry: render expected wrapper content; if target is missing **or differs byte-wise** → atomically install. This means: - - first run after upgrade materializes wrappers for all existing aliases (one-time migration happens on any command, including a harmless `list`); - - every create/edit/remove leaves artifacts consistent by construction; - - hand-edited or half-deleted wrappers are silently healed (idempotent, cheap for realistic alias counts). -2. Every executable in `~/.local/bin` bearing our marker whose name is **not** in ENV → deleted (covers remove, covers manual ENV edits, covers the empty-set case). -3. If ≥1 wrapper exists/installed and `$HOME/.local/bin` is absent from `$PATH` → loud `warn` with copy-paste fix (`export PATH="$HOME/.local/bin:$PATH"` + persist to `~/.profile`). Wrappers are still written regardless. -4. Legacy handling (below). - -No public `sync` subcommand: every subcommand already syncs, so an explicit one adds surface without capability. `POS_SUBCMDS` header stays `create edit remove list show`. - -### Existing `ai-aliases.sh`: stop writing entirely — no deprecated shim - -A shim keeps two activation truths alive, and the sourced-alias-still-shadows-wrapper scenario is precisely the reported failure mode (in interactive bash, aliases take precedence over PATH lookups). The `.sh` artifact must die, not fade. - -On detecting `SH_FILE`, sync emits a warning block explaining that activation moved to `~/.local/bin/` scripts and that stale sourced aliases shadow them until cleaned. Then: - -- **If line 1–3 of the file carry our generator marker** (`Auto-generated by pos ai alias`) → auto-remove the file and print remediation for *running* shells: an `unalias ` line with names extracted from the `.sh` contents themselves (the stale file inventories its own definitions — including names no longer in ENV), plus "or simply start a new shell". Auto-remove is safe because (a) the file is regenerable output, not user data, (b) postinstall never shipped the source line, so nothing references it at startup, and (c) the conditional-source idiom (`[ -f ] && source`) tolerates absence even if a user wired it manually. -- **If the marker does not match** (foreign/hand-built file) → leave untouched; advise manual review. Never delete files we didn't generate — same policy as wrapper collisions. - -*[PROVISIONAL nuance]* The auto-remove guard currently checks only the generator-marker header; a user who appended private aliases into our generated file would lose them on upgrade-migration. Accepted risk: the file header says "do not hand-edit", likelihood is low, and the alternative (parsing full-file provenance) buys complexity the requirement doesn't need. Revisit only if a real case appears. - -### What carries over unchanged - -- `ai-aliases.env` format, location, chmod 600, comment conventions — untouched. Previously created aliases migrate with zero data conversion. -- `pos config` compatibility: no env-key semantics touched. -- All menu flows, name regex, non-tty guard behavior. - -[DECIDED] - -## Decision 3: Edge cases & subcommand semantics - -### Name collisions with real binaries — refuse - -Create-time check order (after existing ENV-duplicate redirect to `edit`): - -1. `$HOME/.local/bin/` exists **with** marker → not a collision; sync will overwrite (regeneration path). -2. `$HOME/.local/bin/` exists **without** marker → refuse: `err "File '~/.local/bin/' already exists and was not created by pos ai alias — pick another name"`. Never silently overwrite foreign files. -3. `command -v ` resolves to anything else on PATH (`ls`, `git`, `gcc`, …) → refuse with the conflicting path named. - -No override flag. Shadowing an arbitrary binary is never a legitimate intent for an *alias* feature, a refusal error costs one rename, and a `--force` surface invites exactly the "surprise factor" this rework is meant to remove. Edit cannot collide (name is the record key); rename remains remove+create (Designer out-of-scope list already excludes renaming). - -### Empty result set → wrappers fully retracted - -With zero ENV entries, sync deletes every marker-bearing wrapper in `~/.local/bin`. `list` prints `Aliases (0):` as today. No empty husks left behind. - -### Provider adapter deleted later → runtime failure is already good enough - -Wrapper execs `pos ai …`; if the adapter vanished, `bin/pos-ai:121` errors: `Unknown provider '

' — available: gemini openrouter`. Actionable, names valid alternatives, zero changes to `pos-ai`. Sync does **not** prune wrappers whose provider directory entry disappeared (ENV is truth for existence; a temporarily missing adapter shouldn't silently eat user config). The provider picker at edit time only offers installed providers, so edit is the natural repair path. - -### Subcommand semantics under Option B - -| Subcommand | Change | -|------------|--------| -| `list` | Format unchanged; runs after sync so it always reflects disk truth | -| `show ` | Adds one line: `Wrapper: ~/.local/bin/` (or `(not installed)` if PATH check failed) | -| `create` | Gains collision refusals above; success message replaces "Reload shell: source ~/.bashrc" with `Available immediately: ~/.local/bin/` (+ PATH warning when applicable) | -| `edit` | Unchanged flow; on save, sync refreshes the wrapper — change is live on next invocation (this kills the reported bug) | -| `remove` | Unchanged confirm(default=n); success message notes the script was deleted from `~/.local/bin`; add hint that running shells may need `hash -r` only if the name still autocompletes stale (rare; bash normally re-scans when a hashed file vanishes) | -| menu / `-h` | Help text updated: activation = executable scripts in `~/.local/bin`, no sourcing required | - -[DECIDED] - ---- - -## Decision 4: Scope fence for Builder - -### Approved outcome -Alias activation via marker-managed wrapper scripts in `~/.local/bin`, synced against `ai-aliases.env` on every invocation, with legacy `.sh` auto-retirement. - -### In-scope components/files -| File | Allowed changes | -|------|-----------------| -| `bin/pos-ai-alias` | Replace `_alias_regen()` with `_wrapper_path()` + `_wrapper_render()` + `_alias_sync()`; keep `_alias_quote_cmd` mechanics verbatim; add `_alias_check_path()`; wire sync into all dispatch entries; collision checks in `_alias_create`; message deltas in create/edit/remove/show/usage; legacy `.sh` retirement block; SH_FILE constant retained solely for migration detection | -| `bin/pos-system-uninstall` | Add marker-scan of `~/.local/bin` (grep for `Managed by pos ai alias`) to the discovery list (~line 96 area) and removal pass (~line 261 area), mirroring the existing `pos-ai-hook.sh` pattern — closes uninstall hygiene | -| `DOC/POS.md` | `ai alias` section: activation semantics, subcommand table unchanged otherwise (hand-maintained file) | -| `DOC/HOWTO.md` / relevant howto | Row/section wording update if it mentions sourcing/reload | -| `AGENT_TODO.md` | Move task to Done (dated) in same commit | - -### Must NOT change -- `bin/pos-ai` — any file byte. -- `ai-aliases.env` format, fields, header comments, chmod 600. -- `lib/common.sh`, `lib/menu-lib.sh`, `lib/config-ui.sh`. -- `postinstall.sh` — the deferred `.bashrc` wiring stays unimplemented by design (obsoleted, not added). -- `# POS:` / `# POS_SUBCMDS:` headers (description and subcommand set unchanged → no gen churn beyond none). -- Menu structure, step counts, name regex, non-tty guard behavior (Designer spec remains authoritative). -- Other categories' tools; entertainment plugins; completions (no flag/subcmd changes). - -### Architectural constraints -1. Atomic writes only (mktemp+mv), never in-place truncation of live wrappers. -2. Ownership established exclusively via the line-2 marker string; never delete/overwrite files failing the marker test. -3. All output discipline per Designer spec: tables/results stdout, display/warnings stderr (`log`/`warn`/`err`). -4. Sync must be idempotent and safe to run concurrently-lossy (single-user tool: last write wins, no locking). -5. Wrapper body contains no secrets and no absolute paths except the `pos` lookup by name (PATH-resolved, consistent with old aliases). - -### Required verification (adversarial where it matters) -1. `bash -n bin/pos-ai-alias`; `make gen && make check && make lint` ending `0 FAIL, 0 WARN`. -2. **Quoting round-trip through the NEW artifact**: prompts containing `'`, `"`, backtick, `$()`, `%`, `\`, unicode, leading/trailing spaces → create each; execute wrapper under a stubbed `pos` shim on a temp PATH capturing argv; assert `--system` arrives as exactly one intact word and passthrough args (`assist "hi there"`) append correctly. -3. **Staleness kill-test**: create `assist`(gemini) → run wrapper via shim → edit provider→openrouter → run again → argv shows openrouter with **no shell reload** (the regression test for the live failure). -4. Sync idempotency: two consecutive runs → byte-identical artifacts, mtimes stable second run. -5. Orphan retraction: delete an ENV line manually → next `pos ai alias list` removes that wrapper; empty ENV → zero owned wrappers remain. -6. Collision tests: foreign file at `~/.local/bin/` → refused; marker file → refreshed; `command -v` conflict (e.g. `gcc`) → refused with path named. -7. Legacy migration: plant prior-generator-format `ai-aliases.sh` with stale `alias assist=…gemini…` → any subcommand removes it, prints `unalias assist` remediation; plant foreign-content file → untouched, warned. -8. PATH-absent: strip `$HOME/.local/bin` from PATH → loud warn, wrappers still written. -9. Non-tty: `pos ai alias` (menu) still fails cleanly via `menu_guard`. -10. Dead-provider runtime: wrapper pointing at removed adapter produces `pos-ai:121` available-providers error (assert message quality manually once). - -### Explicitly out of scope -Rename operation; multi-line prompt input; alias import/export; public `sync` subcommand; completion headers; `pos config` integration; systemd/cron integration examples beyond help text. - -### Open risks -- Users who sourced `ai-aliases.sh` into `.bashrc` manually keep a dead reference — harmless under the conditional-source idiom; warning text covers it. -- `~/.local/bin` absent from PATH in exotic shells (non-login ssh without postinstall) — mitigated by persistent warning + fix line. - -[DECIDED] - ---- - -## Builder-ready step order - -1. Core rewrite in `bin/pos-ai-alias`: `_wrapper_path`, `_wrapper_render` (reuse `_alias_quote_cmd`; marker line 2; `set -euo pipefail`; `exec … "$@"`), `_alias_check_path`, `_alias_sync` (render-diff-install, orphan sweep, legacy block). Delete `_alias_regen` body (keep SH_FILE constant for migration). -2. Wire `_alias_sync` into every dispatch entry before subcommand logic. -3. Create-flow collision refusals (marker-aware, `command -v` check) + success-message swap ("Available immediately", drop reload hints everywhere including menu flows). -4. Edit/remove/show/list/usage deltas per D3 table. -5. `bin/pos-system-uninstall`: marker-scan additions in discovery + removal passes. -6. Full verification suite (D4 list) — quoting round-trip and staleness kill-test are the acceptance gates. -7. Docs (`DOC/POS.md`, HOWTO row) + `AGENT_TODO.md` Done entry; conventional commit (`feat:` or `fix:`). - -Recommended next agent: **Builder** — scope is fully determined; no architectural choices remain. Suggest a Reviewer pass afterward focused on the marker-guard logic (the only place where a bug could delete/overwrite a foreign file). - -Architect changes: this report only. diff --git a/AgentsReport/architect/2026-08-26_registry-architecture.md b/AgentsReport/architect/2026-08-26_registry-architecture.md deleted file mode 100644 index d70b300..0000000 --- a/AgentsReport/architect/2026-08-26_registry-architecture.md +++ /dev/null @@ -1,808 +0,0 @@ -# Architect Report — Self-Describing Command Registry for POS - -**Date:** 2026-08-26 -**Status:** DECISION_READY - ---- - -## TL;DR - -- **Decision:** Add a thin `lib/registry.sh` library that parses `# POS_*:` headers into a queryable API; two new optional headers (`# POS_DEPS:`, `# POS_EXAMPLES:`) extend existing conventions; no framework, no new abstraction layer. -- **Key insight:** The existing header system is already 80% of a registry. The missing piece is a shared parsing library so every consumer stops reimplementing `sed` + `grep` header reading. -- **Scope:** `lib/registry.sh` (new), `scripts/gen-docs.sh` (extend), `bin/pos-tree` (use registry), `templates/pos-tool.sh` (document new headers), docs. All existing tools keep working unchanged. -- **Migration:** Zero-downtime — new headers are optional. Tools add `# POS_DEPS:` and `# POS_EXAMPLES:` incrementally. Registry degrades gracefully when headers are absent. -- **Open items:** `pos help` and `pos menu` are future consumers (not in this phase). Dashboard is out of scope. - ---- - -## Decision 1: Header Format — Extend, Don't Replace - -### Problem - -The spec asks for dependencies, curated examples, and a shared query API. Currently, each consumer (`pos tree`, `pos --help`, `gen-docs.sh`, `pos config`) independently implements `sed`/`grep` header parsing with its own logic. - -### Decision - -Add two new optional `# POS_*:` header lines. Keep all existing headers unchanged. - -### Exact Syntax - -```bash -# POS_DEPS: [binary2 ...] -# POS_EXAMPLES: | -``` - -**Rules:** -- All new headers are **optional** — tools that don't declare them simply won't expose that metadata. Progressive metadata preserved. -- `# POS_DEPS:` is space-separated binary names (what `command -v` checks, not apt package names). -- `# POS_EXAMPLES:` can appear on multiple lines — each is ` | ` (pipe-delimited, max one `|`). -- Headers must appear between the shebang/strict-mode block (lines 1–6) and the first non-comment line. -- Gen-docs and lint only parse headers from the first ~10 lines of each file. - -**Existing headers (unchanged):** - -```bash -# POS: — -# POS_FLAGS: --flag1 --flag2 -# POS_SUBCMDS: sub1 sub2 sub3 -# POS_CONFIG: | | =: | ... -``` - -**New headers (optional):** - -```bash -# POS_DEPS: docker nmap jq -# POS_EXAMPLES: pos network scan 192.168.1.0/24 | Scan a /24 CIDR -# POS_EXAMPLES: pos network scan 10.0.0.0/28 --full | Full scan with OS detection -``` - -### Example: Minimal Tool (No Change Needed) - -```bash -#!/usr/bin/env bash -set -euo pipefail -# POS: ssh load-keys — Load all SSH keys into the agent -# ... rest of script -``` - -### Example: Rich Tool - -```bash -#!/usr/bin/env bash -set -euo pipefail -# POS: network download — aria2 RPC daemon + queue control (add/torrent/metalink, watch, limits) -# POS_SUBCMDS: start stop status add torrent metalink list info files peers pause resume remove purge move limit set watch restart retry replace menu -# POS_FLAGS: --dir --out --split --seed --force --upload --gid --tmux -# POS_DEPS: aria2c jq curl -# POS_EXAMPLES: pos network download add https://example.com/file.zip | Download a file -# POS_EXAMPLES: pos network download status | Show download queue status -# POS_EXAMPLES: pos network download --tmux start aria2 daemon with live view -``` - -### Rationale - -| Option | Architecture | Advantages | Costs | Risks | When to choose | -|--------|-------------|------------|-------|-------|----------------| -| **A: Extend existing headers** | Add `# POS_DEPS:` and `# POS_EXAMPLES:` alongside existing headers | Backward compatible, zero migration cost, progressive metadata, consistent with existing patterns | Two new header formats to parse | Low — optional headers degrade gracefully | **Chosen** — smallest sufficient design | -| B: Unified YAML frontmatter | Replace all `# POS_*:` with a YAML block in each file | Richer structure, easier to extend | Breaks all existing consumers, requires migration of 40+ tools, adds YAML dependency | High — YAML parser availability in bash, migration burden | Only if the header system were fundamentally inadequate | -| C: Separate registry file | `registry/.yaml` per tool | Clean separation, richer metadata | Duplicates what headers already provide, extra files to maintain, sync risk between header and registry | Medium — source of truth drift | Only if headers couldn't hold the metadata | - -**Option A wins** because the existing header system already works, is already the source of truth for gen-docs output, and the new metadata (deps, examples) fits naturally into the comment-header format. - -### Constraints for Builder - -- `# POS_DEPS:` line: `sed -n '/^# POS_DEPS: /{s/^# POS_DEPS: //;p;q}' ` — space-separated tokens. -- `# POS_EXAMPLES:` lines: `grep '^# POS_EXAMPLES:' - Echo sorted tool keys belonging to . - Pass "" for category-less tools. - -# ── Lookup ────────────────────────────────────────────────────── - -reg_lookup - Echo a field's value for a tool. Fields: - cat, desc, flags, subcmds, deps, examples - Returns empty string if field not set or tool not found. - Exit code: 0 if tool found, 1 if not. - -reg_config_scopes - Echo sorted list of unique config scope names. - -reg_config_keys - Echo key|flags|description lines for a scope (newline-delimited). - -reg_config_envfile - Echo the env-file basename for a scope. - Exit code: 0 if found, 1 if not. - -# ── Iteration ─────────────────────────────────────────────────── - -reg_each - Call for each tool, passing: - - Category is empty for category-less tools. - -# ── Convenience (for common patterns) ────────────────────────── - -reg_tool_exists - Exit 0 if tool is registered, 1 otherwise. - -reg_tools_for_category - Alias for reg_tools_in. Kept for clarity. -``` - -### Source Pattern - -```bash -# lib/registry.sh — no shebang (library, not executable) -# Sourced opt-in by consumers that need tool metadata. - -# Common.sh helpers (guarded fallback — mirrors lib/config-ui.sh pattern) -declare -F log >/dev/null || log() { echo "[+] $*"; } -declare -F warn >/dev/null || warn() { echo "[!] $*"; } -declare -F err >/dev/null || err() { echo "ERROR: $*" >&2; exit 1; } - -# ── Tool directory detection ──────────────────────────────────── -# Repo: lib/registry.sh → ../bin -# Install: /usr/local/bin/registry.sh → /usr/local/bin (same dir) -_reg_tools_dir() { - local dir - dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../bin" 2>/dev/null && pwd)" - if [ -d "$dir" ] && ls "$dir"/pos-* &>/dev/null; then - echo "$dir" - else - dirname "${BASH_SOURCE[0]}" - fi -} - -# ── Data stores ───────────────────────────────────────────────── -declare -a _reg_tools=() -declare -A _reg_cat=() -declare -A _reg_desc=() -declare -A _reg_flags=() -declare -A _reg_subcmds=() -declare -A _reg_deps=() -declare -A _reg_examples=() -declare -a _reg_config_scopes=() -declare -A _reg_config_keys=() - -# ── reg_scan ──────────────────────────────────────────────────── -reg_scan() { - local dir="${1:-$(_reg_tools_dir)}" f - local old LC_ALL_PREV="$LC_ALL" - export LC_ALL=C - - _reg_tools=() - # Clear all associative arrays - for key in "${!_reg_cat[@]:-}"; do - unset "_reg_cat[$key]" "_reg_desc[$key]" "_reg_flags[$key]" - unset "_reg_subcmds[$key]" "_reg_deps[$key]" "_reg_examples[$key]" - done - _reg_config_scopes=() - for scope in "${!_reg_config_keys[@]:-}"; do - unset "_reg_config_keys[$scope]" - done - - local -A scope_seen=() - - for f in "$dir"/pos-*; do - [ -x "$f" ] || continue - local name="${f##*/pos-}" - local key cat sub - if [[ "$name" == *-* ]]; then - cat="${name%%-*}" - sub="${name#*-}" - else - cat="" - sub="$name" - fi - key="$sub" - - _reg_tools+=("$key") - _reg_cat["$key"]="$cat" - - # POS: — description (text after first "— ") - local pos_line - pos_line="$(sed -n '/^# POS: /{s/^# POS: //;p;q}' "$f" 2>/dev/null)" - _reg_desc["$key"]="${pos_line#*— }" - - # POS_FLAGS: - _reg_flags["$key"]="$(sed -n '/^# POS_FLAGS: /{s/^# POS_FLAGS: //;p;q}' "$f" 2>/dev/null)" - - # POS_SUBCMDS: - _reg_subcmds["$key"]="$(sed -n '/^# POS_SUBCMDS: /{s/^# POS_SUBCMDS: //;p;q}' "$f" 2>/dev/null)" - - # POS_DEPS: - _reg_deps["$key"]="$(sed -n '/^# POS_DEPS: /{s/^# POS_DEPS: //;p;q}' "$f" 2>/dev/null)" - - # POS_EXAMPLES: (may appear multiple times — join with newlines) - local examples="" - examples="$(sed -n '/^# POS_EXAMPLES: /{s/^# POS_EXAMPLES: //;p}' "$f" 2>/dev/null)" - _reg_examples["$key"]="$examples" - - # POS_CONFIG: (may appear multiple lines per file) - local line - while IFS= read -r line; do - [ -n "$line" ] || continue - line="${line#*POS_CONFIG:}" - local scope="${line%%|*}" - scope="${scope// }" - [ -n "$scope" ] || continue - _reg_config_keys["$scope"]+="${_reg_config_keys[$scope]:+$'\n'}$line" - if [ -z "${scope_seen[$scope]:-}" ]; then - scope_seen["$scope"]=1 - _reg_config_scopes+=("$scope") - fi - done < <(grep '^# POS_CONFIG:' "$f" 2>/dev/null || true) - done - - # Sort tools - mapfile -t _reg_tools < <(printf '%s\n' "${_reg_tools[@]}" | sort) - # Sort config scopes - mapfile -t _reg_config_scopes < <(printf '%s\n' "${_reg_config_scopes[@]}" | sort -u) - - export LC_ALL="$LC_ALL_PREV" -} - -# ── Discovery ─────────────────────────────────────────────────── -reg_list() { printf '%s\n' "${_reg_tools[@]}"; } - -reg_categories() { - local -A cats=() - local t - for t in "${_reg_tools[@]}"; do - cats["${_reg_cat[$t]}"]=1 - done - printf '%s\n' "${!cats[@]}" | sort -} - -reg_tools_in() { - local cat="$1" t - for t in "${_reg_tools[@]}"; do - [ "${_reg_cat[$t]}" = "$cat" ] && echo "$t" - done -} - -# ── Lookup ────────────────────────────────────────────────────── -reg_lookup() { - local tool="$1" field="$2" - case "$field" in - cat) echo "${_reg_cat[$tool]:-}" ;; - desc) echo "${_reg_desc[$tool]:-}" ;; - flags) echo "${_reg_flags[$tool]:-}" ;; - subcmds) echo "${_reg_subcmds[$tool]:-}" ;; - deps) echo "${_reg_deps[$tool]:-}" ;; - examples) echo "${_reg_examples[$tool]:-}" ;; - *) return 1 ;; - esac -} - -reg_config_scopes() { printf '%s\n' "${_reg_config_scopes[@]}"; } - -reg_config_keys() { - local scope="$1" - echo "${_reg_config_keys[$scope]:-}" -} - -reg_config_envfile() { - local scope="$1" line - line="$(echo "${_reg_config_keys[$scope]:-}" | head -1)" - [ -n "$line" ] || return 1 - line="${line#*|}" # drop scope - local env="${line%%|*}" - echo "${env// }" -} - -# ── Iteration ─────────────────────────────────────────────────── -reg_each() { - local cb="$1" t - for t in "${_reg_tools[@]}"; do - "$cb" "${_reg_cat[$t]}" "$t" "${_reg_desc[$t]}" - done -} - -# ── Convenience ───────────────────────────────────────────────── -reg_tool_exists() { - [ -n "${_reg_desc[$1]+x}" ] -} -``` - -### Rationale - -| Option | Architecture | Advantages | Costs | Risks | When to choose | -|--------|-------------|------------|-------|-------|----------------| -| **A: Regenerate-on-source library** | `reg_scan` parses all files into bash arrays on first call | O(1) lookups after scan, no external deps, works from /usr/local/bin, bash-native | ~180 lines of code, scan cost at startup (~5ms for 40 tools) | Low — scan is fast enough for interactive use | **Chosen** — matches project's bash-only, no-framework philosophy | -| B: Cached JSON file | `make gen` produces `registry.json`, consumers parse with `jq` | Fast lookups, rich queries | Requires `jq` at runtime (currently a dep, but adds coupling), extra build step, staleness risk | Medium — JSON dependency for all consumers | Only if performance of header parsing became a bottleneck (it won't for 40 tools) | -| C: Per-tool .meta files | Each tool has a sidecar `pos-.meta` | Clean separation, rich format | Duplicates header data, extra files to maintain, sync drift risk | Medium — two sources of truth | Only if headers were fundamentally limited | - -**Option A wins** because: -1. Headers are already the source of truth — no sync risk. -2. The scan takes ~5ms for 40 tools — no performance concern. -3. Works from `/usr/local/bin/` (all libs live there after install). -4. Matches the project's "no unnecessary framework" philosophy. -5. Follows the pattern established by `lib/config-ui.sh`. - -### Source Chain - -```bash -# In any consumer: -source "$(dirname "$0")/../lib/registry.sh" 2>/dev/null || source "$(dirname "$0")/registry.sh" -reg_scan -# ... use reg_list, reg_lookup, etc. -``` - -[DECIDED] - ---- - -## Decision 3: Consumer Integration - -### Problem - -Four consumers need to use the registry. Each has different requirements: -- `bin/pos-tree` needs tree-building from tool data -- `bin/pos` `_pos_category_help()` needs category → tool listing -- `scripts/gen-docs.sh` needs all metadata for code generation -- `lib/config-ui.sh` needs config scopes and keys - -### Decision - -Migrate consumers to `lib/registry.sh` in this order (highest value first): - -#### 3.1: `bin/pos-tree` (Priority: HIGH) - -**Current state** (lines 47–68): Scans `pos-*` files independently, reads `# POS:` and `# POS_SUBCMDS:` via `sed`. - -**After migration:** -```bash -source "$(dirname "$0")/../lib/registry.sh" 2>/dev/null || source "$(dirname "$0")/registry.sh" -reg_scan - -# Build tree from registry instead of scanning files -for tool in $(reg_list); do - cat="$(reg_lookup "$tool" cat)" - desc="$(reg_lookup "$tool" desc)" - deps="$(reg_lookup "$tool" deps)" - # ... add to tree, annotate with deps if present -done -``` - -**Enhancement:** When `# POS_DEPS:` is present, show it in the tree view: -``` -├── docker ps # Enhanced container overview (health, IPs, ports, uptime) -│ [deps: docker] -``` - -This is the highest-value consumer change — it proves the registry works at runtime and shows the immediate benefit of new metadata. - -**Implementation note:** Keep the existing `add()` / `render()` tree-building logic. Replace only the data-collection loop (lines 47–68) with registry calls. The tree structure is already correct from filenames + `POS_SUBCMDS`. - -#### 3.2: `scripts/gen-docs.sh` (Priority: HIGH) - -**Current state** (lines 30–46): Collects tools into a pipe-delimited array via direct `sed` calls. - -**After migration:** -```bash -# In the tools collection loop, use registry for new fields -# Keep the existing collection pattern for backward compatibility -# (gen-docs.sh has its own sorting and rendering logic) - -# Add deps and examples to the tools array format -tools+=("$cat|$sub|$desc|$flags|$subcmds|$deps|$examples") -``` - -**New gen blocks:** -- Add `deps` and `examples` columns to `gen_tree()`, `gen_dispatch()`, `gen_filetable()`. -- These become visible in `DOC/AGENT_Context_Project.md` once tools add the new headers. - -**Important:** The existing `gen_*` functions use their own `tools` array (not the registry) because they need specific formatting. The registry provides the raw data; gen-docs formats it. This avoids tight coupling between the generator and the library. - -**Alternative considered:** Have gen-docs source the registry directly. Rejected because gen-docs needs the data in a specific format (pipe-delimited array) and the registry's data structure is an implementation detail. Keeping the data flow explicit (`registry → gen-docs tools array → gen_* functions`) is cleaner. - -#### 3.3: `bin/pos` `_pos_category_help()` (Priority: MEDIUM) - -**Current state** (lines 68–123): Scans `pos--*` files, reads `# POS:` and `# POS_SUBCMDS:` per file. - -**After migration:** -```bash -_pos_category_help() { - local cat="$1" - source "$(dirname "$0")/../lib/registry.sh" 2>/dev/null || source "$(dirname "$0")/registry.sh" - reg_scan - - echo "pos $cat — $cat tools" - echo - echo "USAGE" - echo " pos $cat [args]" - echo - echo "COMMANDS" - for tool in $(reg_tools_in "$cat"); do - local desc deps - desc="$(reg_lookup "$tool" desc)" - deps="$(reg_lookup "$tool" deps)" - printf ' %-28s%s' "$tool" "$desc" - [ -n "$deps" ] && printf ' [deps: %s]' "$deps" - echo - # ... subcommands from reg_lookup "$tool" subcmds - done - echo - echo "Run 'pos $cat --help' for details on a command." - exit 0 -} -``` - -**Trade-off:** This adds a `reg_scan` call every time `pos ` runs. For 40 tools, the scan takes ~5ms — negligible for interactive use. If profiling shows this matters, `pos` could cache the scan in a temp file (but this is premature optimization). - -**Alternative considered:** Keep `_pos_category_help()` using direct `sed` (no registry dependency). Rejected because the whole point is to centralize header parsing. The 5ms scan cost is acceptable. - -#### 3.4: `lib/config-ui.sh` (Priority: LOW — separate decision) - -**Current state** (lines 50–203): Has its own header parsing for `# POS_CONFIG:` and `# POS_KEYS:`. The config library is already a mature, working abstraction. - -**Decision: Do NOT integrate config-ui.sh with the registry in this phase.** - -**Rationale:** -1. config-ui.sh already works correctly and is well-tested. -2. Its parsing is specialized (multi-line key fields, `*plugins` expansion, `*providers` expansion). -3. Integrating it with the registry would require the registry to handle all config-ui's edge cases, bloating the library. -4. config-ui.sh's `cfg_headers()` / `cfg_scopes()` / `cfg_scope_keys()` API is already the "registry" for config consumers. - -**Future:** When config-ui.sh needs maintenance, it could source the registry for its initial scan. But this is not needed now. - -### Rationale - -| Option | Architecture | Advantages | Costs | Risks | When to choose | -|--------|-------------|------------|-------|-------|----------------| -| **A: Incremental migration** | Migrate tree + gen-docs first, then pos, defer config-ui | Lowest risk, proves value early, no breaking changes | Some consumers still use direct sed during transition | Low — transition period is harmless | **Chosen** — smallest risky step | -| B: Big-bang migration | Rewrite all consumers at once | Consistent from day one | High risk, many things can break, hard to review | High — one bad refactor breaks everything | Only if the codebase were much smaller | -| C: No migration — just add the library | Create registry.sh but don't change any consumers | Library exists for future use | No immediate value, consumers still duplicate logic | Low — but pointless | Only if the task were just "create a library" | - -[DECIDED] - ---- - -## Decision 4: Migration Strategy — Zero-Downtime Incremental - -### Problem - -40+ tools exist. All must keep working. The new headers are optional. No tool should require changes to function. - -### Decision - -**Phase 1 (this implementation):** -1. Create `lib/registry.sh` with full API. -2. Update `scripts/gen-docs.sh` to parse new headers (degrades gracefully when absent). -3. Migrate `bin/pos-tree` to use registry (proves runtime value). -4. Update `templates/pos-tool.sh` to document new headers. -5. Update `DOC/DEV.md` and `DOC/AGENT_Context_Project.md` with new header format. - -**Phase 2 (future, out of scope):** -1. Migrate `bin/pos` `_pos_category_help()` to use registry. -2. Enhance `pos help` to show registry metadata (deps, examples). -3. Add `# POS_DEPS:` and `# POS_EXAMPLES:` to tools incrementally (start with 3–5 representative tools per category). - -**Phase 3 (future, out of scope):** -1. `pos menu` — interactive menu from registry (new tool). -2. `pos dashboard` — status dashboard from registry (new tool). - -### Incremental Adoption for Tool Authors - -1. Add `# POS_DEPS: docker jq` to your tool's header block. Done — deps appear in registry. -2. Add `# POS_EXAMPLES: pos | Description` lines. Done — examples appear in registry. -3. No other changes required. The tool keeps working exactly as before. -4. When you run `make gen`, the new metadata appears in generated docs. - -### Tool Template Update - -`templates/pos-tool.sh` gets new header documentation: - -```bash -# ──────────────────────────────────────────────────────────────── -# TEMPLATE — new `pos` CLI tool -# -# 1. Copy: cp templates/pos-tool.sh bin/pos-- -# 2. Header: add a `# POS:` line right after the shebang/strict-mode -# lines (single source of truth for generated docs): -# # POS: — one-line description -# # POS_FLAGS: --flag1 --flag2 (flag-style tools only) -# # POS_SUBCMDS: sub1 sub2 (multi-command tools only) -# # POS_DEPS: binary1 binary2 (runtime deps, optional) -# # POS_EXAMPLES: pos | Description (optional) -# 3. ... -``` - -### Lint Gate Update - -`scripts/lint-conventions.sh` gets a new WARN check: -- If `# POS_DEPS:` is present, validate that each token looks like a binary name (no spaces, no special chars). This is a soft check — WARN on malformed deps, not FAIL. - -[DECIDED] - ---- - -## Decision 5: Scope Boundaries - -### Approved Scope - -```text -In scope: - lib/registry.sh — NEW FILE, ~180 lines - scripts/gen-docs.sh — Extend tools array, add deps/examples to gen_* functions - bin/pos-tree — Migrate to use registry (replace file-scanning loop) - templates/pos-tool.sh — Document new headers in template comments - DOC/DEV.md — Document new header format and registry usage - DOC/AGENT_Context_Project.md — Update line count table for lib/registry.sh, gen blocks updated by make gen - scripts/lint-conventions.sh — Optional: WARN for malformed POS_DEPS - -Not in scope: - bin/pos _pos_category_help() — Phase 2 (future) - pos help enhancements — Phase 2 (future) - pos menu (new tool) — Phase 3 (future) - pos dashboard (new tool) — Phase 3 (future) - Adding POS_DEPS/POS_EXAMPLES to existing tools — Individual tool authors, incremental - lib/config-ui.sh integration — Deferred (already works, specialized parsing) - completions/pos.bash changes — No new completion data needed (deps/examples aren't completable) - install.sh changes — registry.sh is installed with existing lib/* loop -``` - -### What Builder Must NOT Do - -1. Do NOT add `# POS_DEPS:` or `# POS_EXAMPLES:` to any existing tool in this PR (that's incremental migration, separate commits). -2. Do NOT change the `# POS:` header format or the em-dash convention. -3. Do NOT change `bin/pos` dispatch logic or `INTERACTIVE_CMDS`. -4. Do NOT add any new files beyond `lib/registry.sh`. -5. Do NOT modify the completion script (`completions/pos.bash`). -6. Do NOT refactor `lib/config-ui.sh` to use the registry. -7. Do NOT add a shebang to `lib/registry.sh` (it's a library, not executable). -8. Do NOT break `make gen && make check && make lint`. - -[DECIDED] - ---- - -## Decision 6: Risk Analysis - -### Risk 1: `make gen` output changes break `git diff --exit-code` in CI - -**Impact:** HIGH — blocks PRs. - -**Cause:** Adding deps/examples parsing to gen-docs.sh changes the `tools` array format. The gen_* functions that consume this array will produce different output if any tool has the new headers. But since no tools have them yet, the output should be identical. - -**Mitigation:** -- The gen_* functions must produce **identical output** when no tools have `# POS_DEPS:` or `# POS_EXAMPLES:` headers. -- Test: run `make gen && git diff --exit-code` before committing. Zero diff = safe. -- The tools array format change (adding `$deps|$examples` fields) is internal to gen-docs.sh — the gen_* functions that render output must not use the new fields when they're empty. - -### Risk 2: `reg_scan` performance degrades with many tools - -**Impact:** LOW — current tool count is ~40. - -**Cause:** Each tool file is read 4–6 times by `sed` during scan. For 40 tools, this is ~200 process spawns. - -**Mitigation:** -- At current scale, scan completes in ~5ms. Even 100 tools would be ~15ms. -- If it ever becomes an issue, `reg_scan` could read each file once and parse all headers in a single `awk` pass. This is a future optimization, not needed now. - -### Risk 3: Registry library conflicts with existing sourcing patterns - -**Impact:** MEDIUM — could break tools that source both common.sh and registry.sh. - -**Cause:** `registry.sh` declares guarded fallbacks for `log`, `warn`, `err` (same pattern as `config-ui.sh`). If both are sourced, the second source is a no-op because the functions already exist. - -**Mitigation:** -- Use the same guarded-declaration pattern as `config-ui.sh`: `declare -F log >/dev/null || log() { ... }`. -- `registry.sh` does NOT define `run`, `spawn`, `confirm`, or any other common.sh functions. -- `registry.sh` does NOT call `err` during normal operation — only if `reg_scan` is called with an invalid directory (which won't happen in practice). - -### Risk 4: `bin/pos-tree` migration breaks tree output - -**Impact:** HIGH — visible to users. - -**Cause:** The tree-building logic in `pos-tree` is tightly coupled to the current data collection. Replacing the collection loop might subtly change tree structure. - -**Mitigation:** -- Keep the existing `add()`, `render()` functions unchanged. -- Replace ONLY the data-collection loop (lines 47–68) with registry calls. -- Test: run `bin/pos-tree` before and after, diff the output. Must be identical (for existing headers). -- The only visible change should be when a tool has `# POS_DEPS:` — deps appear in the tree. - -### Risk 5: `pos ` performance regression - -**Impact:** LOW — adds ~5ms per invocation. - -**Cause:** `_pos_category_help()` would source and call `reg_scan` on every invocation. - -**Mitigation:** -- Phase 2 only (not in this implementation). -- If needed, cache scan results in a temp file: `reg_scan` writes to `/tmp/.pos-registry-` and consumers check for freshness. This is premature — implement only if profiling shows a problem. - -### Risk 6: New headers malformed, breaking parsing - -**Impact:** LOW — malformed headers produce empty values, not crashes. - -**Cause:** A tool author writes `# POS_DEPS` (missing colon) or `# POS_EXAMPLES foo bar` (missing pipe). - -**Mitigation:** -- `reg_scan` uses strict pattern matching: `sed -n '/^# POS_DEPS: /{...}'`. Missing colon = no match = empty value. Graceful degradation. -- Add a lint WARN (not FAIL) for `# POS_DEPS:` lines without space-separated tokens. -- Add lint WARN for `# POS_EXAMPLES:` lines without `|` delimiter. -- Document the expected format clearly in DEV.md. - -[DECIDED] - ---- - -## Verification Plan - -### Gate 1: Syntax - -```bash -bash -n lib/registry.sh # Must pass (no syntax errors) -bash -n scripts/gen-docs.sh # Must pass (after modifications) -bash -n bin/pos-tree # Must pass (after modifications) -``` - -### Gate 2: Gen Drift - -```bash -make gen && git diff --exit-code # Zero diff (no tools have new headers yet) -``` - -### Gate 3: Self-Consistency - -```bash -make check # Must pass (syntax + exec bits + doc/code sync + dispatch smoke) -``` - -### Gate 4: Convention Lint - -```bash -make lint # Must pass (0 FAIL, 0 WARN) -``` - -### Gate 5: Functional - -```bash -# Registry works standalone -bash -c 'source lib/registry.sh; reg_scan; reg_list; reg_lookup docker ps desc' - -# pos tree produces identical output -bin/pos-tree > /tmp/tree-before.txt -# ... apply changes ... -bin/pos-tree > /tmp/tree-after.txt -diff /tmp/tree-before.txt /tmp/tree-after.txt # Must be empty - -# pos category help works -bin/pos docker --help # Must show docker tools -bin/pos network --help # Must show network tools -``` - -### Gate 6: Regression - -```bash -# All existing commands still dispatch -bin/pos --help -bin/pos docker --help -bin/pos network --help -bin/pos help network scan -``` - ---- - -## Implementation Guidance for Builder - -### Step-by-step - -1. **Create `lib/registry.sh`** (~180 lines). Start from the API spec in Decision 2. Use `lib/config-ui.sh` as a structural reference for the guarded fallbacks and source pattern. - -2. **Update `scripts/gen-docs.sh`**. In the tools collection loop (lines 30–46): - - Add `deps` and `examples` fields to the `tools` array format: `"$cat|$sub|$desc|$flags|$subcmds|$deps|$examples"` - - Parse new headers with the same `sed` pattern as existing ones. - - In `gen_tree()`, `gen_dispatch()`, `gen_filetable()`: add deps/examples columns ONLY when non-empty. Empty fields = identical output to current. - -3. **Migrate `bin/pos-tree`**. Replace lines 47–68 (the file-scanning loop) with registry calls. Keep `add()`, `render()`, and the rest unchanged. Test that output is identical for existing tools. - -4. **Update `templates/pos-tool.sh`**. Add `# POS_DEPS:` and `# POS_EXAMPLES:` to the header documentation block. Add them after the existing `# POS_FLAGS:` example. - -5. **Update `DOC/DEV.md`**. In "Adding a New CLI Tool → Make it discoverable": - - Document the new `# POS_DEPS:` and `# POS_EXAMPLES:` headers. - - Explain when to use each (deps: list runtime binaries; examples: show 1–3 representative usages). - -6. **Update `lib/registry.sh` line count** in `DOC/AGENT_Context_Project.md` filetable (the hand-maintained rows above the GEN marker). - -7. **Run gates:** `make gen && make check && make lint`. Verify 0 FAIL, 0 WARN. - -### Critical Constraints - -- `lib/registry.sh` must NOT have a shebang (library, not executable). -- `lib/registry.sh` must be added to the `lib_names` list in `install.sh` Phase 2 (line 143). -- `reg_scan` must set `LC_ALL=C` for deterministic sort. -- gen-docs.sh changes must produce zero diff when no tools have new headers. -- `pos-tree` output must be byte-identical before/after migration (for existing headers). - ---- - -## Open Questions (for Orchestrator) - -1. Should `pos --help` (in `bin/pos`) be migrated in this phase or deferred to Phase 2? **Recommendation: defer to Phase 2** — lower risk, and the category help is already working. - -2. Should the lint gate enforce that `# POS_DEPS:` tokens are valid binary names? **Recommendation: WARN only, not FAIL** — some deps might be shell builtins or paths, not just binary names. - -3. Should `reg_scan` support a `--cached` mode? **Recommendation: no, not yet** — premature optimization for 40 tools. - ---- - -*Report written by Architect agent. Next recommended agent: **Builder** (to implement the approved scope).* diff --git a/AgentsReport/architect/2026-09-04_hf-downloader-architecture.md b/AgentsReport/architect/2026-09-04_hf-downloader-architecture.md deleted file mode 100644 index 40360e2..0000000 --- a/AgentsReport/architect/2026-09-04_hf-downloader-architecture.md +++ /dev/null @@ -1,572 +0,0 @@ -# Architecture Report — `pos ai hf` (Hugging Face Model Downloader) - -**Date:** 2026-09-04 -**Status:** DECISION_READY - ---- - -## TL;DR - -| Decision | Choice | Rationale | -|----------|--------|-----------| -| File location | `bin/pos-ai-hf` | Subcommand of `ai` category — models are AI infrastructure | -| Subcommands | `download`, `search`, `list`, `remove` | Core + discovery + local management | -| Config scope | `ai` (existing) | No new scope needed — `HF_TOKEN` and `HF_DOWNLOAD_DIR` fit the existing `ai.env` | -| Default download dir | `~/.local/share/linux_post_install/ai/models/` | Follows XDG data conventions, matches `SESSION_DIR` parent | -| Dependencies | `curl`, `jq` | Already in `preinstall.sh` PACKAGES — no changes | -| Progress | curl `--progress-bar` | Native, no extra deps, works for multi-GB files | -| Resume | curl `-C -` | Automatic resume on interrupted downloads | -| New files created | `bin/pos-ai-hf` (1 file) | Minimal scope — everything else is doc updates | -| Open items | HOWTO.md + howto/ai.md updates (Writer task) | Not blocking implementation | - ---- - -## Decision 1: File Location — `bin/pos-ai-hf` - -**Problem:** Where does a Hugging Face model downloader live in the `pos` hierarchy? - -**Evidence:** -- Existing AI tools: `bin/pos-ai` (692 lines), `bin/pos-ai-alias` (760 lines), `bin/pos-ai-gemini`/`bin/pos-ai-openrouter` (7-line forwarders) — all under the `ai` category -- The `ai` category covers AI providers, sessions, models, and assistants -- Downloading models is AI infrastructure — it feeds ollama, llama.cpp, and similar local inference tools -- The user's explicit context: "download AI models for local inference" - -**Options:** - -| Option | Architecture | Pros | Cons | -|--------|-------------|------|------| -| A: `bin/pos-ai-hf` | Subcommand of `ai` category | Consistent with existing AI tool hierarchy; `pos ai hf download` is natural; `pos ai --help` shows it alongside other AI tools | Slightly longer invocation path | -| B: `bin/pos-ai-download` | Named after the action, not the provider | Action-first naming | Conflates "AI download" with "HF download"; would need renaming when adding other model sources (e.g., CivitAI, Ollama registry) | -| C: `bin/pos-hf` | Own category | Shortest invocation | Breaks `ai` category coherence; HF is not a general tool category | - -**Decision:** Option A — `bin/pos-ai-hf` - -**Reasoning:** HF is a provider/source within the AI domain. The `ai` category already contains provider-specific tools (`pos-ai-gemini`, `pos-ai-openrouter`). Adding `pos-ai-hf` for model downloading fits this pattern perfectly. The tool name communicates both the domain (`ai`) and the source (`hf`). - -**Convention compliance:** -- `# POS: ai hf — Download AI models from Hugging Face (search, download, manage)` -- No new category in `pos --help` -- Auto-discovered by `pos` dispatcher - -[DECIDED] - ---- - -## Decision 2: Subcommands - -**Problem:** What operations should `pos ai hf` support? - -**Evidence:** -- `pos media grab` pattern (file:83-141): thin classifier + delegator — minimal surface area -- `pos network download` (18 subcommands): comprehensive but for a complex download manager with queuing, torrents, retry logic -- `pos ai` pattern (file:1-10): flag-based with subcommands (`ask`, `chat`, `sessions`, `models`, `providers`) -- User goal: "Download AI models for local inference" — primary operation is download; search and local management are secondary - -**Options:** - -| Option | Subcommands | Pros | Cons | -|--------|-------------|------|------| -| A: download + search + list + remove | 4 subcommands | Full lifecycle; covers discovery, download, local management, cleanup | More surface area to maintain | -| B: download + list + remove | 3 subcommands | Core + local management; search can be done via `curl` manually | User must leave `pos` for discovery | -| C: download only | 1 subcommand | Minimal; simplest to implement and maintain | No local management; user must track paths manually | - -**Decision:** Option A — `download`, `search`, `list`, `remove` - -**Reasoning:** -- `download` is the primary operation (user goal) -- `search` is low-cost to implement (one API call, jq formatting) and high-value for discovery -- `list` shows what's already downloaded — essential for a model management workflow -- `remove` lets users clean up without manually tracking paths -- Total surface area is manageable — each subcommand is a single function, not a complex state machine - -**Subcommand contracts:** - -``` -pos ai hf download [filename] # Download a file or entire repo -pos ai hf download --files # List files, then download selected -pos ai hf download --branch # Download from a specific branch/commit -pos ai hf download --gguf # Download only .gguf files (inference-ready) -pos ai hf search # Search HF models -pos ai hf list # List downloaded models -pos ai hf remove # Remove a downloaded model -``` - -[DECIDED] - ---- - -## Decision 3: Config Scope — Extend `ai.env` - -**Problem:** Where do `HF_TOKEN` and `HF_DOWNLOAD_DIR` live? - -**Evidence:** -- `# POS_CONFIG:` header format: `scope | file | KEY=:description | ...` -- Existing `ai` scope: `bin/pos-ai` line 6 — `# POS_CONFIG: ai | ai.env | AI_PROVIDER=…` -- HF download is an AI tool — its config logically belongs with other AI config -- Creating a new `hf` scope would add another `pos config` entry and `.env` file for just 2 keys -- The `pos config ai` command already exists and users would expect AI-related config there - -**Options:** - -| Option | Architecture | Pros | Cons | -|--------|-------------|------|------| -| A: Extend `ai.env` (existing scope) | `HF_TOKEN` and `HF_DOWNLOAD_DIR` added to `# POS_CONFIG: ai` in `bin/ai-hf` | One config location for all AI tools; user runs `pos config ai` to see everything | Mixes provider keys (GEMINI_API_KEY) with download config | -| B: New `hf.env` (new scope) | `# POS_CONFIG: hf | hf.env | HF_TOKEN=…` | Clean separation; `pos config hf` is self-contained | Another `.env` file; users must know which scope has the token | -| C: `system.env` (shared scope) | `HF_TOKEN` and `HF_DOWNLOAD_DIR` in system.env via `load_system_env()` | Centralizes shared config | Wrong semantic — HF is AI-specific, not system-wide | - -**Decision:** Option A — Extend existing `ai` scope - -**Reasoning:** -- The `ai` scope already holds `AI_PROVIDER` and `AI_GEMINI_API_KEY` — adding HF keys keeps all AI config in one place -- Users run `pos config ai` once to configure everything they need for AI tools -- No new scope registration, no new `.env` file, no new completion entry -- The `# POS_CONFIG:` header in `bin/pos-ai-hf` adds its keys to the same `ai` scope - -**Header addition (in `bin/pos-ai-hf`):** -```bash -# POS_CONFIG: ai | ai.env | HF_TOKEN=:Hugging Face API token (https://huggingface.co/settings/tokens) (secret) | HF_DOWNLOAD_DIR=:Model download directory (default ~/.local/share/linux_post_install/ai/models) -``` - -[DECIDED] - ---- - -## Decision 4: Download Directory Layout - -**Problem:** Where do downloaded files land, and what directory structure? - -**Evidence:** -- `SESSION_DIR="$HOME/.local/share/linux_post_install/ai"` (bin/pos-ai line 12) — data convention for AI tools -- `DOWNLOAD_DIR="${DOWNLOAD_DIR:-$HOME/Downloads}"` (bin/pos-network-download line 22) — general download convention -- Ollama expects models in `~/.ollama/models/` — not our concern (user moves files) -- llama.cpp uses `--model ` — just needs the path printed -- HF repos use `/` format (e.g., `meta-llama/Llama-3.1-8B-Instruct`) - -**Decision:** Flat layout under XDG data directory - -``` -~/.local/share/linux_post_install/ai/models/ -├── meta-llama-Llama-3.1-8B-Instruct/ -│ ├── config.json -│ ├── model.safetensors -│ ├── tokenizer.json -│ └── .hf-meta # our metadata: repo-id, branch, download date, files -├── Qwen-Qwen2.5-7B-Instruct/ -│ ├── model-00001-of-00003.safetensors -│ └── ... -└── TheBloke-Mistral-7B-v0.1-GGUF/ - ├── mistral-7b-v0.1.Q4_K_M.gguf - └── .hf-meta -``` - -**Key decisions:** -- **Folder name:** `-` (dash-joined, slashes replaced). Clean, filesystem-safe, human-readable. -- **Default base:** `~/.local/share/linux_post_install/ai/models/` (overridable via `HF_DOWNLOAD_DIR`) -- **`.hf-meta` file:** JSON metadata (repo-id, branch, download timestamp, file list). Enables `list` and `remove` without API calls. -- **No nesting by namespace:** Flat is simpler — users can see all models at a glance. - -**Out of scope:** Integrating with ollama's model directory or llama.cpp's model directory. Users move files themselves or use `--output` flag. - -[DECIDED] - ---- - -## Decision 5: Download Logic - -**Problem:** How to download files from HF repos reliably. - -**Evidence:** -- HF REST API: `GET /api/models/{ns}/{repo}` returns file list (`siblings[].rfilename`) -- `GET /api/models/{ns}/{repo}/tree/{rev}/{path}` returns sizes + LFS info -- Large files (7GB+): curl `-L` transparently handles LFS, Xet, CDN redirects -- Rate limits: 500/5min anonymous, 1000/5min with token -- No single "download all" endpoint — must loop through file list -- `curl -C -` handles resume for interrupted downloads -- `--progress-bar` gives native progress for large files - -**Download flow:** - -``` -1. Validate repo-id (must contain /) -2. Call GET /api/models/{ns}/{repo} → extract siblings -3. Filter files (by filename arg, --gguf flag, or download all) -4. For each file: - a. Create target directory (mkdir -p) - b. Construct download URL: https://huggingface.co/{ns}/{repo}/resolve/{rev}/{filename} - c. curl -L -C - --progress-bar -H "Authorization: Bearer $HF_TOKEN" → target - d. Verify file exists and is non-empty -5. Write .hf-meta (repo-id, branch, files, timestamp) -6. Print summary: path, total size, file count -``` - -**Key implementation details:** - -| Concern | Solution | -|---------|----------| -| Auth | Always pass `Authorization: Bearer $HF_TOKEN` header — even public repos get better rate limits | -| Large files | `curl -L` handles LFS/Xet transparently; `--progress-bar` shows native progress | -| Resume | `curl -C -` resumes interrupted downloads automatically | -| Rate limiting | Sleep 1s between files; on 429, wait `Retry-After` header value or 60s default | -| Disk space | Pre-flight check: `df` available space vs estimated total (from `/tree/` endpoint) | -| Partial download | If curl fails mid-file, the partial file remains (resume on next run) | -| Gated repos | API returns 403 without token; with valid token, same flow works | - -**File listing (for `--files` flag):** -``` -GET /api/models/{ns}/{repo}/tree/main/ | jq to extract filenames + sizes -``` - -**Progress for multi-file downloads:** -- Single file: curl `--progress-bar` is sufficient -- Multi-file: Print `[N/M]` counter before each file's download, curl `--progress-bar` for each - -**Avoided complexity:** -- No aria2 dependency (pos-network-download pattern) — curl is sufficient for sequential downloads -- No parallel downloads — complexity not justified for single-user homelab use -- No streaming/progress tracking library — curl's native progress bar is enough - -[DECIDED] - ---- - -## Decision 6: Output Contract - -**Problem:** What does the tool print? - -**Evidence:** -- `pos media grab` prints: emoji + title + path + size (file:202-226) -- `pos network download add` prints: GID for tracking -- User goal: "Print the path so the user knows where files landed" -- Tool output goes to stdout (captured by `tee` for logging) - -**Decision:** Structured, parseable output with human-friendly summary - -``` -# Single file download: -📥 Downloaded: meta-llama/Llama-3.1-8B-Instruct/model.safetensors (4.7 GB) -📁 ~/.local/share/linux_post_install/ai/models/meta-llama-Llama-3.1-8B-Instruct/model.safetensors - -# Multi-file download: -📥 Downloaded: meta-llama/Llama-3.1-8B-Instruct (7 files, 4.7 GB) -📁 ~/.local/share/linux_post_install/ai/models/meta-llama-Llama-3.1-8B-Instruct/ - -# Search results: -Found 5 models for "llama 7b": - meta-llama/Llama-2-7b-chat-hf 12.3k downloads 13.5 GB - NousResearch/Llama-2-7b-hf 8.2k downloads 13.5 GB - ... - -# List: -Downloaded models (3): - meta-llama-Llama-3.1-8B-Instruct 4.7 GB 2026-09-04 - Qwen-Qwen2.5-7B-Instruct 4.2 GB 2026-09-03 - TheBloke-Mistral-7B-v0.1-GGUF 4.1 GB 2026-09-02 - -# Remove: -Removed: meta-llama-Llama-3.1-8B-Instruct (freed 4.7 GB) -``` - -**stdout contract:** -- Summary lines go to stdout (logged by `tee`) -- Progress bars go to stderr (not logged) -- Errors go to stderr via `err()` (exits 1) - -[DECIDED] - ---- - -## Decision 7: Error Handling - -**Problem:** How to handle failure modes gracefully. - -**Evidence:** -- `pos network download` has comprehensive error handling for RPC failures, dead sources, outages -- `pos media grab` has URL validation and delegation failure summary -- `err()` from common.sh exits 1 with red message -- Network tools need to handle transient failures - -**Error matrix:** - -| Error | Detection | Response | -|-------|-----------|----------| -| Missing deps | `command -v` guard before `--help` | `err "curl not found (install curl)"` — exits before help | -| Invalid repo format | No `/` in repo-id | `err "Invalid repo format: use namespace/model-name"` | -| 404 (repo not found) | HTTP status from API | `err "Model not found: {repo-id}"` | -| 401/403 (auth) | HTTP status | `err "Authentication failed — check HF_TOKEN (pos config ai)"` | -| 429 (rate limit) | HTTP status | Sleep `Retry-After` or 60s, retry once, then fail | -| Network timeout | curl exit code 28 | `err "Connection timed out — check network"` | -| Disk space | `df` pre-flight | `warn "Low disk space: need {N} GB, only {M} GB available"` then continue (user's call) | -| Partial download | curl exit code != 0 | `warn "Download interrupted for {file} (resume with same command)"` — partial file stays | -| jq parse error | jq exit code | `err "Failed to parse API response — check network or HF status"` | -| Token not set | Empty after config load | `warn "No HF_TOKEN set — using anonymous access (lower rate limits)"` — continue for public repos | - -**Design principle:** Never fail silently. Always tell the user what happened and how to fix it. For transient errors, offer resume guidance. - -[DECIDED] - ---- - -## Decision 8: Dependencies and Lint Compliance - -**Problem:** What deps are needed, and how to satisfy the lint gate? - -**Evidence:** -- `curl` and `jq` are in `preinstall.sh` PACKAGES (line 29: `git curl wget aria2 vim nano tmux tree jq`) -- Deps guards must sit **before** `-h|--help` case (DEV.md line 110, lint rule) -- `set -euo pipefail` required (lint rule) -- `# POS:` header required (lint rule) -- No stdin reading → not in `INTERACTIVE_CMDS` (lint rule) - -**Lint compliance checklist:** - -| Rule | Requirement | Implementation | -|------|-------------|----------------| -| Shebang | `#!/usr/bin/env bash` | Line 1 | -| Strict mode | `set -euo pipefail` | Line 2 | -| `# POS:` header | After shebang/strict-mode | Lines 3-7 | -| Deps guards before `--help` | `command -v` guards before case | After source, before case | -| `-h|--help` via case | `case "${1:-}" in -h\|--help) usage ;;` | Standard pattern | -| Exec bits | 100755 | `chmod +x` on creation | -| No stdin | Not in `INTERACTIVE_CMDS` | True — non-interactive tool | -| Source chain | `source "$(dirname "$0")/../lib/common.sh" 2>/dev/null \|\| source "$(dirname "$0")/common.sh"` | Standard pattern | - -**No new packages needed.** `curl` and `jq` are already installed by `preinstall.sh`. - -[DECIDED] - ---- - -## Decision 9: Testing Strategy - -**Problem:** How to verify the tool works without a live HF token or network. - -**Evidence:** -- DEV.md (line 196-214): Stub PATH approach — fake binaries, temp HOME, assert on output -- `pos network download` test pattern: fake curl/systemctl stubs with JSON fixtures -- `pos system backup` test pattern: per-test lsblk JSON fixtures in temp dirs -- Env-overridable paths: `HF_DOWNLOAD_DIR` is the seam - -**Test architecture:** - -``` -/tmp/opencode/hf-test/ -├── run-tests.sh # Test runner with check() helper -├── stubs/ -│ ├── curl # Fake curl: returns fixtures based on URL pattern -│ └── jq # Pass-through (real jq with fixture data) -└── fixtures/ - ├── model-meta.json # GET /api/models/{ns}/{repo} response - ├── model-tree.json # GET /api/models/{ns}/{repo}/tree/ response - └── search.json # GET /api/models?search=... response -``` - -**Test cases (target: ~40-50 cases):** - -| Category | Cases | -|----------|-------| -| Argument parsing | Missing repo-id, invalid format (no /), unknown subcommand, unknown flag | -| download | Single file download, whole repo download, --gguf filter, --branch, resume (partial file exists), 404 error, 401 error, 429 rate limit | -| search | Successful search, empty results, network error | -| list | Empty list, populated list, corrupted .hf-meta | -| remove | Successful remove, nonexistent model, remove frees space | -| Config | Token from env, token from file, download dir override | -| Output | Summary format matches contract, paths are correct | -| Edge cases | Empty repo, very long filename, special characters in repo-id | - -**Stub `curl` behavior:** -- Intercepts calls to `huggingface.co` -- Routes `/api/models/` to fixture files -- Routes `/resolve/` to a fake download (creates a small file) -- Simulates error codes (401, 403, 404, 429) -- Tracks call count for assertion - -**No changes to the repo's test infrastructure** — tests live in `/tmp/opencode/` per convention. - -[DECIDED] - ---- - -## Function Signatures - -### Config - -```bash -load_hf_config() -# Reads HF_TOKEN and HF_DOWNLOAD_DIR from: -# 1. Already-exported env vars (highest precedence) -# 2. ~/.config/linux_post_install/ai.env (HF_TOKEN, HF_DOWNLOAD_DIR) -# 3. Defaults: HF_DOWNLOAD_DIR=~/.local/share/linux_post_install/ai/models -``` - -### API Helpers - -```bash -hf_api() # hf_api → JSON response (GET only) -# Calls: curl -fsS -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/api$endpoint" -# Handles: 401/403 auth errors, 429 rate limit (sleep + retry once), network errors - -hf_repo_files() # hf_repo_files [branch] → JSON array of {rfilename, size} -# Calls: GET /api/models/{ns}/{repo}/tree/{branch}/ for sizes, falls back to /api/models/{ns}/{repo} for file list - -hf_search() # hf_search [limit] → JSON array of {id, downloads, likes} -# Calls: GET /api/models?search={query}&sort=downloads&direction=-1&limit={N} -``` - -### Download - -```bash -hf_download_file() # hf_download_file → 0/1 -# curl -L -C - --progress-bar -H "Authorization: Bearer $HF_TOKEN" -o "$target" "$url" -# Returns: 0 on success, 1 on curl failure - -hf_download_repo() # hf_download_repo [branch] [filename|--gguf] -# Orchestrates: API call → file list → loop → download → write .hf-meta → summary -``` - -### Subcommands - -```bash -cmd_download() # cmd_download [args...] -# Dispatches: single file / whole repo / --files interactive / --gguf filter - -cmd_search() # cmd_search -# Calls hf_search, formats table - -cmd_list() # cmd_list -# Scans $HF_DOWNLOAD_DIR, reads .hf-meta, prints table - -cmd_remove() # cmd_remove -# Validates exists, rm -rf, prints freed space -``` - -### Utilities - -```bash -hf_repo_dir() # hf_repo_dir → filesystem path (dash-joined) -# "meta-llama/Llama-3.1-8B-Instruct" → "$HF_DOWNLOAD_DIR/meta-llama-Llama-3.1-8B-Instruct" - -hf_human_size() # hf_human_size → "4.7 GB" / "12.3 MB" / "1024 B" -# Same pattern as pos-media-grab (file:213-221) - -hf_resolve_branch() # hf_resolve_branch [branch] → resolved branch -# Default "main"; calls API to get model metadata defaultBranch if not specified -``` - ---- - -## Config Keys - -| Key | Scope | File | Default | Secret | Description | -|-----|-------|------|---------|--------|-------------| -| `HF_TOKEN` | `ai` | `ai.env` | (empty) | Yes | Hugging Face API token. Generate at huggingface.co/settings/tokens. Even for public repos, a token increases rate limits from 500/5min to 1000/5min. | -| `HF_DOWNLOAD_DIR` | `ai` | `ai.env` | `~/.local/share/linux_post_install/ai/models` | No | Base directory for downloaded models. Each repo gets a subdirectory named `-`. | - -**Precedence:** env var > `ai.env` file > default (standard `load_config` pattern). - ---- - -## File List and Responsibilities - -### New files - -| File | Purpose | Lines (est.) | -|------|---------|-------------| -| `bin/pos-ai-hf` | Main tool: download, search, list, remove | ~350-400 | - -### Modified files - -| File | Change | Scope | -|------|--------|-------| -| `DOC/POS.md` | Add `ai hf` to `ai` category table + detail block | Hand-written | -| `DOC/HOWTO.md` | Add row to AI category in index | Hand-written | -| `DOC/howto/ai.md` | Add Hugging Face download section (recipes, config, troubleshooting) | Hand-written | -| `DOC/AGENT_Context_Project.md` | GEN blocks auto-regenerated by `make gen` | Auto | - -### NOT modified - -| File | Reason | -|------|--------| -| `preinstall.sh` | `curl` and `jq` already in PACKAGES | -| `lib/common.sh` | No shared helpers needed — tool is self-contained | -| `bin/pos` | No `INTERACTIVE_CMDS` change (non-interactive tool); usage EXAMPLES updated by hand | -| `install.sh` | No new lib files to install | - ---- - -## Approved Scope - -### In scope - -1. **Create `bin/pos-ai-hf`** — single file, ~350-400 lines - - Subcommands: `download`, `search`, `list`, `remove` - - Config: `HF_TOKEN`, `HF_DOWNLOAD_DIR` via `# POS_CONFIG: ai` - - Deps guards for `curl` and `jq` - - Full `--help` text - - Error handling for all failure modes listed in Decision 7 - - Resume support (`curl -C -`) - - Rate limit handling (sleep + retry on 429) - - `.hf-meta` metadata tracking per downloaded repo - -2. **Documentation updates** (Writer task, not blocking) - - `DOC/POS.md`: `ai hf` row + detail block - - `DOC/howto/ai.md`: Hugging Face section - - `DOC/HOWTO.md`: index row - -3. **Run gates** - - `make gen && make check && make lint` must pass (0 FAIL, 0 WARN) - -### Explicitly out of scope - -- **Ollama integration** — no `ollama import` or model registration; user moves files manually -- **llama.cpp integration** — no quantization or conversion; just download -- **Parallel downloads** — sequential is sufficient for homelab use -- **Download queuing/history** — no aria2 dependency; simple curl-based downloads -- **Model conversion** — pure download tool, not a model pipeline -- **CivitAI/other sources** — HF only; other sources get their own tools if needed -- **Interactive file picker** — `--files` lists files and downloads all (or filtered); no interactive selection menu -- **New config scope** — extends existing `ai` scope, no new `pos config` entry -- **`pos ai` changes** — `bin/pos-ai` is not modified; `pos-ai-hf` is independent - ---- - -## Architectural Constraints for Builder - -1. **Start from template:** `cp templates/pos-tool.sh bin/pos-ai-hf` -2. **POS header must be on line ~4:** `# POS: ai hf — Download AI models from Hugging Face (search, download, manage)` -3. **Deps guards before `-h|--help` case:** `command -v curl` and `command -v jq` before the case -4. **Source chain:** Standard `source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"` -5. **Config loader pattern:** Copy `load_config()` from `bin/pos-ai` (lines 130-160) — read `ai.env`, env-var precedence, strip CR -6. **All file paths must be seam-guarded:** `HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}"` -7. **Output to stdout only:** Summary lines. Progress bars and curl output to stderr. -8. **No INTERACTIVE_CMDS change:** Tool does not read stdin -9. **`make gen && make check && make lint` must pass** before handoff to Writer - ---- - -## Verification - -| Check | Command | Expected | -|-------|---------|----------| -| Syntax | `bash -n bin/pos-ai-hf` | No output, rc=0 | -| Exec bit | `ls -la bin/pos-ai-hf` | `-rwxr-xr-x` | -| POS header | `head -10 bin/pos-ai-hf` | Contains `# POS: ai hf —` | -| Help | `bin/pos-ai-hf --help` | Prints usage, rc=0 | -| Deps guard | Remove curl, run `bin/pos-ai-hf --help` | Error about curl, rc=1 | -| Dispatch | `bin/pos help ai hf` | Shows pos-ai-hf help | -| Category | `bin/pos ai --help` | Lists `hf` subcommand | -| Gen | `make gen` | Regenerates tree/dispatch/completions | -| Check | `make check` | 0 failures | -| Lint | `make lint` | 0 FAIL, 0 WARN | -| Stub tests | `/tmp/opencode/hf-test/run-tests.sh` | 40+ cases green | - ---- - -## Risks - -| Risk | Likelihood | Impact | Mitigation | -|------|-----------|--------|------------| -| HF API changes endpoint format | Low | Medium | Pin to v0 API (`/api/models/`); monitor HF changelog | -| Token exposure in logs | Medium | High | Token passed via header, not URL; never printed in output; `HF_TOKEN` marked as secret in POS_CONFIG | -| Very large repos (100+ files) | Low | Low | Sequential download with progress; user can Ctrl+C and resume | -| LFS pointer files downloaded instead of content | Low | Medium | `curl -L` follows LFS redirect; test with known LFS repo | -| Disk full during multi-file download | Medium | Medium | Pre-flight `df` check; partial files preserved for resume | diff --git a/AgentsReport/architect/2026-09-04_llamacpp-server-architecture.md b/AgentsReport/architect/2026-09-04_llamacpp-server-architecture.md deleted file mode 100644 index 38fd4d8..0000000 --- a/AgentsReport/architect/2026-09-04_llamacpp-server-architecture.md +++ /dev/null @@ -1,533 +0,0 @@ -# Architecture Decision: `pos ai server` — llama.cpp Inference Server - -## TL;DR - -**Decisions:** -1. New tool `bin/pos-ai-server` with subcommands: `start`, `stop`, `status`, `models`, `logs` -2. Systemd user service generated at runtime (same pattern as `pos-network-download` and `pos-communication-telegram-listener`) -3. Config extends existing `ai` scope in `ai.env` — no new config files -4. Provider adapter `lib/ai-providers/llamacpp.sh` follows the 4-function contract -5. Changes to `bin/pos-ai`: `resolve_key()` accepts `llamacpp` (no key needed), `resolve_model()` falls through to `LLAMACPP_MODEL`, `cmd_providers()` includes llamacpp -6. No static `systemd/` unit file — the service is generated dynamically because model path, port, and GPU flags are user-configurable - -**Open items:** -- llama-server binary name varies by build (`llama-server`, `llama.cpp/server`, `server`) — detection logic needs a fallback chain -- ROCm detection deferred (Debian/Ubuntu focus, CUDA-only auto-detect) - ---- - -## Decision 1: Tool Structure - -**Problem:** User needs to start/stop/manage a local llama.cpp inference server via `pos`. - -**Decision:** Create `bin/pos-ai-server` as a standalone tool under the `ai` category, with subcommands. - -**Evidence:** -- Existing pattern: `bin/pos-network-download` is a standalone tool with `start`/`stop`/`status` subcommands for the aria2 daemon -- Existing pattern: `bin/pos-communication-telegram-listener` manages its own systemd user service -- Tool naming: `pos-ai-server` → `pos ai server` (category: `ai`, command: `server`) - -**Subcommands:** - -| Subcommand | Description | -|------------|-------------| -| `start [model]` | Install & start the systemd user service (model from arg, config, or interactive pick) | -| `stop` | Stop & remove the service | -| `status` | Show running state, loaded model, port, health endpoint | -| `models` | List available GGUF files from `HF_DOWNLOAD_DIR` | -| `logs [lines]` | Show recent server logs via `journalctl --user` | - -**Not interactive:** `pos ai server` does NOT read stdin (no prompts that block under `tee`). It does NOT need to be in `INTERACTIVE_CMDS`. - -**Approved scope:** -- `bin/pos-ai-server` — 1 file -- POS header: `# POS: ai server — llama.cpp local inference server (start, stop, status, models, logs)` -- POS_SUBCMDS: `start stop status models logs` -- POS_DEPS: `curl jq` (curl for health check + API, jq for JSON parsing) -- POS_FLAGS: `--port --host --model --ctx --gpu --threads` - -[DECIDED] - ---- - -## Decision 2: Systemd User Service (Runtime-Generated) - -**Problem:** The llama-server service needs model path, port, GPU layers, and other parameters that are user-configurable. A static unit file can't carry these. - -**Decision:** Generate the systemd user service file at runtime (same pattern as `pos-network-download` lines 172-187 and `pos-communication-telegram-listener` lines 481-500). - -**Evidence:** -- `pos-network-download`: generates `pos-aria2.service` at `cmd_start()` with `$RPC_PORT`, `$RPC_SECRET`, `$DOWNLOAD_DIR` baked into `ExecStart` -- `pos-communication-telegram-listener`: generates `pos-telegram-listener.service` with the runner path baked in -- Both write to `$USER_SYSTEMD_DIR` (`~/.config/systemd/user/`), then `systemctl --user daemon-reload && enable --now` -- Both use `cat > "$USER_SYSTEMD_DIR/$SERVICE" < --port --host --n-gpu-layers --ctx-size --threads -Restart=on-failure -RestartSec=5 -TimeoutStopSec=10 -KillMode=control-group -EnvironmentFile=-%h/.config/linux_post_install/ai.env - -[Install] -WantedBy=default.target -``` - -**Key design choices:** -- `Type=simple` — llama-server runs in foreground by default (no daemonize flag needed) -- `Restart=on-failure` — restart if it crashes, but not on clean exit (`stop` sends SIGTERM, which is clean) -- `RestartSec=5` — give time for model unload/reload -- `TimeoutStopSec=10` — llama-server handles SIGTERM gracefully (unloads model), 10s is generous -- `KillMode=control-group` — ensures the whole process tree is cleaned up -- `EnvironmentFile=-` (dash prefix) — missing file is not an error -- ExecStart is a direct `llama-server` call (not a wrapper script) — systemd handles the lifecycle - -**Where config values come from:** `cmd_start()` reads the config file, resolves all values, then bakes them into the generated unit. The `EnvironmentFile` line in the unit is a fallback but the actual arguments are baked in at generation time. This matches the aria2 pattern exactly. - -**`start` subcommand flow:** -1. Load config from `ai.env` -2. Resolve model (argument → `LLAMACPP_MODEL` → interactive pick) -3. Resolve port, host, ctx, gpu, threads (flag → config → default) -4. Validate model file exists -5. Auto-detect GPU if `LLAMACPP_GPU_LAYERS` is `-1` -6. Check port availability -7. Generate systemd unit file -8. `systemctl --user daemon-reload` -9. `systemctl --user enable --now pos-ai-server.service` -10. Wait briefly, then check health endpoint - -**Linger warning:** Same as existing tools — warn if `loginctl enable-linger` is needed. - -[DECIDED] - ---- - -## Decision 3: Config Keys (Extend `ai` Scope) - -**Problem:** Server settings need to be persisted alongside existing AI config. - -**Decision:** Extend the existing `ai` scope in `ai.env`. No new config file. - -**Evidence:** -- `ai.env` already holds `AI_PROVIDER`, `AI_GEMINI_API_KEY`, `HF_DOWNLOAD_DIR`, etc. -- The `# POS_CONFIG:` header on `bin/pos-ai` already declares the `ai` scope -- Adding `LLAMACPP_*` keys to the same file keeps all AI config in one place -- `pos config ai` auto-discovers keys from `# POS_CONFIG:` headers - -**Config keys to add:** - -| Key | Default | Description | -|-----|---------|-------------| -| `LLAMACPP_PORT` | `8088` | Server listen port | -| `LLAMACPP_HOST` | `127.0.0.1` | Bind address | -| `LLAMACPP_MODEL` | *(empty)* | Default model path (GGUF file) | -| `LLAMACPP_CTX_SIZE` | `4096` | Context window size | -| `LLAMACPP_GPU_LAYERS` | `-1` | GPU layers (`-1` = auto-detect, `0` = CPU only) | -| `LLAMACPP_THREADS` | `$(nproc)` | CPU threads | - -**POS_CONFIG header on `pos-ai`:** Extend the existing `# POS_CONFIG:` line to include the new keys. The existing header already uses `ai | ai.env | ...` format — we append `LLAMACPP_*` entries. - -**New header addition (appended to existing `# POS_CONFIG:` line):** -``` -| LLAMACPP_PORT=:Server port (default 8088) | LLAMACPP_HOST=:Bind address (default 127.0.0.1) | LLAMACPP_MODEL=:Default model path (GGUF) | LLAMACPP_CTX_SIZE:num:Context window size (default 4096) | LLAMACPP_GPU_LAYERS:num:GPU layers (-1=auto, 0=CPU only, default -1) | LLAMACPP_THREADS:num:CPU threads (default: nproc) -``` - -**Config template update:** Add commented examples to `config/ai.env`. - -[DECIDED] - ---- - -## Decision 4: Provider Adapter - -**Problem:** `pos ai ask` should work with the local llama.cpp server as a backend, just like gemini/openrouter. - -**Decision:** Create `lib/ai-providers/llamacpp.sh` with the 4-function contract. - -**Evidence:** -- `lib/ai-providers/gemini.sh` and `lib/ai-providers/openrouter.sh` both implement: `provider_name()`, `provider_default_model()`, `provider_generate()`, `provider_models_list()` -- `pos-ai` loads providers via `load_provider()` which sources `$PROVIDER_DIR/$p.sh` -- The provider adapter pattern is established and stable - -**Function signatures:** - -```bash -# provider_name → human-readable name -provider_name() { printf 'Local llama.cpp'; } - -# provider_default_model → what's loaded on the server -provider_default_model() { - local port="${LLAMACPP_PORT:-8088}" - local model - model="$(curl -sf "http://127.0.0.1:$port/v1/models" 2>/dev/null | jq -r '.data[0].id // empty')" - [ -n "$model" ] && printf '%s' "$model" || printf '(no model loaded)' -} - -# provider_generate($1=model, $2=messages JSON, $3=optional system prompt) -# → POST to /v1/chat/completions, stdout = response text -provider_generate() { - local model="$1" messages="$2" system="${3:-}" port="${LLAMACPP_PORT:-8088}" - local body resp code body_out - # Build messages array with optional system prompt - if [ -n "$system" ]; then - body="$(printf '%s' "$messages" | jq -c --arg s "$system" \ - '[{role:"system",content:$s}] + .messages')" - else - body="$(printf '%s' "$messages" | jq -c '.messages')" - fi - body="$(printf '%s' "$body" | jq -nc --arg m "$model" --argjson msgs "$body" \ - '{model:$m, messages:$msgs, stream:false}')" - resp="$(curl -sS -m 120 -X POST "http://127.0.0.1:$port/v1/chat/completions" \ - -H "Content-Type: application/json" \ - --write-out $'\n%{http_code}' \ - --data "$body")" || { echo "request failed (curl exit $?)" >&2; return 1; } - code="${resp##*$'\n'}" - body_out="${resp%$'\n'*}" - if [ "$code" != "200" ]; then - echo "API error $code" >&2 - return 1 - fi - printf '%s' "$body_out" | jq -r '.choices[0].message.content // ""' -} - -# provider_models_list($1=current model) → stdout = formatted list -provider_models_list() { - local model="$1" port="${LLAMACPP_PORT:-8088}" resp code body - resp="$(curl -sf "http://127.0.0.1:$port/v1/models" \ - --write-out $'\n%{http_code}')" || { echo "server not running" >&2; return 1; } - code="${resp##*$'\n'}" - body="${resp%$'\n'*}" - [ "$code" = "200" ] || { echo "API error $code" >&2; return 1; } - echo "Local llama.cpp models:" - printf '%s' "$body" | jq -r '.data[]? | .id' | while IFS= read -r m; do - [ -n "$m" ] || continue - if [ "$m" = "$model" ]; then - printf ' %-48s <- loaded\n' "$m" - else - printf ' %-48s\n' "$m" - fi - done -} -``` - -**No API key:** `resolve_key()` in `bin/pos-ai` needs a `llamacpp)` case that succeeds without a key. The local server has no auth. - -**Provider detection in `resolve_key()`:** -```bash -llamacpp) return 0 ;; # No API key needed for local server -``` - -**Provider detection in `resolve_model()`:** -```bash -llamacpp) - [ -n "${LLAMACPP_MODEL:-}" ] && printf '%s' "$(basename "$LLAMACPP_MODEL")" && return ;; -``` - -**Provider detection in `cmd_providers()`:** The existing loop over `$PROVIDER_DIR/*.sh` auto-discovers `llamacpp.sh`. The `configured` check needs updating — llamacpp is "configured" when `llama-server` is available, not when an API key exists. - -**POS_CONFIG header on adapter:** Add `# PROVIDER_CONFIG: LLAMACPP_MODEL=:Default model path (GGUF file)` to the adapter so `pos config ai` discovers it. - -[DECIDED] - ---- - -## Decision 5: GPU Auto-Detection - -**Problem:** Auto-detect NVIDIA CUDA to set `--n-gpu-layers` appropriately. - -**Decision:** Simple CUDA detection — check `nvidia-smi` and `/dev/nvidia*`. No ROCm for now (Debian/Ubuntu focus). - -**Evidence:** -- `nvidia-smi` is the standard NVIDIA management interface -- `/dev/nvidia*` devices indicate driver presence -- llama.cpp uses `--n-gpu-layers -1` for auto (offload all possible layers to GPU) -- The user's request says "Debian/Ubuntu so mainly CUDA" - -**Detection function (in `pos-ai-server`):** - -```bash -detect_gpu() { - if command -v nvidia-smi &>/dev/null && nvidia-smi &>/dev/null 2>&1; then - echo "cuda" - else - echo "cpu" - fi -} -``` - -**GPU layer resolution:** - -```bash -resolve_gpu_layers() { - local configured="${LLAMACPP_GPU_LAYERS:-}" - if [ -n "$configured" ] && [ "$configured" != "-1" ]; then - echo "$configured" - return - fi - # Auto-detect - local gpu - gpu="$(detect_gpu)" - case "$gpu" in - cuda) echo "-1" ;; - *) echo "0" ;; - esac -} -``` - -**Behavior:** -- `LLAMACPP_GPU_LAYERS=-1` (default) → auto-detect: CUDA → `-1`, no CUDA → `0` -- `LLAMACPP_GPU_LAYERS=0` → CPU only (override) -- `LLAMACPP_GPU_LAYERS=35` → explicit layer count (for fine-tuning) - -**No ROCm:** Explicitly out of scope. The detection function can be extended later. - -[DECIDED] - ---- - -## Decision 6: Model Selection - -**Problem:** User needs to pick from downloaded GGUF models. - -**Decision:** `pos ai server models` scans `HF_DOWNLOAD_DIR` for `.gguf` files, reusing `pos ai hf list` patterns. - -**Evidence:** -- `pos-ai-hf` downloads to `$HF_DOWNLOAD_DIR` (default `~/.local/share/linux_post_install/ai/models/`) -- `cmd_list()` in `pos-ai-hf` already scans for model directories with `.hf-meta` files -- GGUF files are the inference-ready format; they're the only files that matter for serving - -**`models` subcommand behavior:** - -```bash -cmd_models() { - local dir="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}" - [ -d "$dir" ] || { warn "No models directory — run 'pos ai hf download' first"; return 0; } - - local found=0 - echo "Available GGUF models:" - while IFS= read -r gguf; do - [ -f "$gguf" ] || continue - found=1 - local name size - name="$(basename "$gguf")" - local dir_name - dir_name="$(basename "$(dirname "$gguf")")" - size="$(stat -c%s "$gguf" 2>/dev/null || echo 0)" - local human_size - human_size="$(human_size "$size")" - printf ' %-50s %s %s\n' "$dir_name/$name" "$human_size" "" - done < <(find "$dir" -name '*.gguf' -type f 2>/dev/null | sort) - - [ "$found" -eq 0 ] && warn "No .gguf files found — download with 'pos ai hf download --gguf'" -} -``` - -**Model resolution order for `start [model]`:** -1. Explicit argument: `pos ai server start /path/to/model.gguf` -2. Relative path argument: `pos ai server start model.gguf` → search `HF_DOWNLOAD_DIR` -3. Config: `LLAMACPP_MODEL` from `ai.env` -4. Interactive pick: prompt user to select from available models - -**Interactive pick (only when on a TTY and no model specified):** -```bash -pick_model() { - local models=() i - while IFS= read -r f; do - [ -f "$f" ] || continue - models+=("$f") - done < <(find "$HF_DOWNLOAD_DIR" -name '*.gguf' -type f 2>/dev/null | sort) - - [ ${#models[@]} -gt 0 ] || err "No GGUF models found — run 'pos ai hf download --gguf'" - - echo "Available models:" - for ((i = 0; i < ${#models[@]}; i++)); do - local name size - name="$(basename "${models[$i]}")" - size="$(stat -c%s "${models[$i]}" 2>/dev/null || echo 0)" - printf ' %2d) %-50s %s\n' "$((i + 1))" "$name" "$(human_size "$size")" - done - echo - local choice - printf 'Pick a model [1-%d]: ' "${#models[@]}" - IFS= read -r choice /dev/null)" || { echo "not running"; return 1; } - # llama.cpp /health returns {"status": "ok"} or {"status": "loading model", ...} - local status - status="$(printf '%s' "$resp" | jq -r '.status // "unknown"' 2>/dev/null)" - echo "$status" -} -``` - -**`logs` subcommand:** Uses `journalctl --user -u pos-ai-server -n --no-pager`. - -[DECIDED] - ---- - -## Decision 8: Changes to `bin/pos-ai` - -**Problem:** `pos ai --provider llamacpp` should work, routing through the local server. - -**Decision:** Minimal changes to `bin/pos-ai` — 3 touch points. - -**Evidence:** -- `resolve_key()` has a `case "$p" in` that checks each provider — add `llamacpp)` case -- `resolve_model()` has a `case "$p" in` for provider-specific fallbacks — add `llamacpp)` case -- `cmd_providers()` checks API key configuration — add llamacpp case -- `require_key()` has provider-specific error messages — add llamacpp case - -**Changes:** - -1. **`resolve_key()` (line 167):** Add `llamacpp) return 0 ;;` — no key needed -2. **`resolve_model()` (line 193):** Add `llamacpp) [ -n "${LLAMACPP_MODEL:-}" ] && printf '%s' "$(basename "$LLAMACPP_MODEL")" && return ;;` -3. **`cmd_providers()` (line 120):** Add `llamacpp) [ -n "${LLAMACPP_PORT:-}" ] && configured="configured" || configured="configured (default port)" ;;` — local server is always "configured" -4. **`require_key()` (line 176):** Add `llamacpp) ;;` — no key needed, just return - -These are all 1-2 line additions within existing `case` blocks. - -[DECIDED] - ---- - -## Decision 9: Error Handling - -| Error | Detection | Response | -|-------|-----------|----------| -| `llama-server` not found | `command -v llama-server` fails | `err "llama-server not found — install llama.cpp (https://github.com/ggerganov/llama.cpp)"` | -| Port in use | `ss -tlnp` or `curl` to port | `err "Port $PORT already in use — check with 'ss -tlnp'"` | -| Model not found | `[ -f "$model" ]` | `err "Model not found: $model"` | -| GPU not detected | `detect_gpu` returns `cpu` | `warn "No NVIDIA GPU detected — running in CPU mode"` (continues) | -| Service start fails | `systemctl --user start` returns non-zero | `journalctl --user -u pos-ai-server -n 20 --no-pager` | -| Server unhealthy | `/health` returns non-200 or times out | `warn "Server may not be ready yet — check with 'pos ai server status'"` | -| Model too large | Not reliably detectable pre-load | Skip — llama.cpp will fail with OOM and the error is in journal logs | - -**Binary detection fallback:** llama.cpp builds name the binary differently: - -```bash -find_llamacpp() { - local candidates=("llama-server" "llama.cpp/server" "server" "llama-server-cuda") - for bin in "${candidates[@]}"; do - command -v "$bin" &>/dev/null && { echo "$bin"; return 0; } - done - return 1 -} -``` - -[DECIDED] - ---- - -## Decision 10: File List & Responsibilities - -| File | Action | Responsibility | -|------|--------|----------------| -| `bin/pos-ai-server` | **NEW** | Service manager: start/stop/status/models/logs, systemd unit generation, GPU detection, model selection | -| `lib/ai-providers/llamacpp.sh` | **NEW** | Provider adapter: provider_name, provider_default_model, provider_generate, provider_models_list | -| `bin/pos-ai` | **MODIFY** | Add `llamacpp` cases to resolve_key, resolve_model, cmd_providers, require_key | -| `config/ai.env` | **MODIFY** | Add commented LLAMACPP_* key documentation | -| `completions/pos.bash` | **AUTO** | `make gen` picks up new POS headers — no manual edit | - -**NOT in scope:** -- No static `systemd/pos-ai-server.service` file (generated at runtime) -- No changes to `postinstall.sh` (service is user-managed, not installed by system) -- No changes to `bin/pos` dispatcher (tool is auto-discovered) -- No changes to `lib/common.sh` - ---- - -## Decision 11: Implementation Constraints - -1. **All `LLAMACPP_*` config reads must go through `load_config()`** — the existing config loader in `pos-ai` (line 131). The new tool also needs its own config loader (or sources `pos-ai`'s, which it can't cleanly). **Decision:** `pos-ai-server` uses its own `load_config()` copy (same pattern as `pos-ai-hf` line 26 — every tool that reads `ai.env` has its own loader). - -2. **The systemd unit must NOT hardcode HOME.** The existing pattern (`pos-communication-telegram-listener` line 494-496) explains why: "Do NOT pin Environment=HOME here — the systemd user manager already sets the correct HOME." - -3. **ExecStart must use full paths for llama-server** — systemd user services don't inherit the user's full `$PATH`. Resolve via `$(command -v llama-server)` at unit generation time. - -4. **The service must use `--log-format` flag** if available (llama.cpp) to produce parseable logs. Not a hard requirement. - -5. **`make gen` must run after creating `bin/pos-ai-server`** to regenerate the tree, dispatch table, completions, and doc tables. - -6. **The tool must pass `make check && make lint`** — bash -n syntax, exec bit, POS header, --help, deps guards before help. - ---- - -## Verification Plan - -1. **Unit test (stub PATH):** - - Fake `llama-server`, `nvidia-smi`, `curl`, `jq` in PATH - - Assert `cmd_start` generates correct unit file content - - Assert `cmd_models` finds `.gguf` files - - Assert `detect_gpu` logic - - Assert config resolution precedence (flag > env > default) - -2. **Integration test (manual):** - - `pos ai server start model.gguf` with a real llama-server binary - - `pos ai server status` shows correct info - - `pos ai server logs` shows journal output - - `pos ai server stop` cleans up - - `pos ai --provider llamacpp ask "hello"` routes through local server - -3. **Gates:** - - `make check` — green - - `make lint` — 0 FAIL, 0 WARN - ---- - -## Explicitly Out of Scope - -- Model conversion/quantization -- Multi-GPU support -- Authentication on the API endpoint -- Web UI -- GPU driver installation -- ROCm/AMD detection -- Quantization awareness (context size vs model capability) -- Model memory estimation / pre-flight checks -- Automatic model download on `start` if none present diff --git a/AgentsReport/architect/2026-09-04_media-grab-architecture.md b/AgentsReport/architect/2026-09-04_media-grab-architecture.md deleted file mode 100644 index d0a6d46..0000000 --- a/AgentsReport/architect/2026-09-04_media-grab-architecture.md +++ /dev/null @@ -1,544 +0,0 @@ -# Architect Report — `pos media grab` + Telegram URL Routing - -**Date:** 2026-09-04 -**Status:** DECISION_READY - ---- - -## TL;DR - -- **Decision 1:** New tool `bin/pos-media-grab` — domain-based URL classifier that delegates to `pos media mp3`/`pos media mp4`, adds `--best` for non-interactive Telegram context. -- **Decision 2:** Listener gains URL routing step between prefix map and AI bridge — detects bare URLs, forwards to `pos media grab`. -- **Decision 3:** Config scope `grab` (`grab.env`) for `GRAB_DEFAULT` mode. -- **Decision 4:** No INTERACTIVE_CMDS change — grab is fully non-interactive; yt-dlp progress suppressed in favor of clean summary. -- **Decision 5:** 600s timeout for downloads in the listener context. -- **Decision 6:** Stub-based test harness in `/tmp/opencode/media-grab-test/`. - ---- - -## Decision 1: File Location — Tool, Not Feature - -### Problem - -Where does `pos media grab` live? `bin/pos-media-grab` (auto-discovered tool) or `features/media-grab.sh` (user-customizable, never-overwritten feature)? - -### Evidence - -- `bin/pos-media-grab` gets auto-discovery via `pos` dispatcher, `# POS:` headers, `make gen` doc tables, completion. -- `features/` scripts are for user-customizable logic (currently: `autostart.sh`, `usb-automount.sh`). They are installed on demand via `./install.sh --feature` and never overwritten on install. -- `pos media grab` is core media routing — it must be present by default, not opt-in. - -### Decision - -`bin/pos-media-grab` — standard tool. - -### Rationale - -Core routing behavior that the listener depends on. Not user-customizable. Fits the `pos--` naming exactly. - -### In Scope - -- `bin/pos-media-grab` (new tool) -- Listener change in `bin/pos-communication-telegram-listener` (add URL routing step) - -### Explicitly Out of Scope - -- Feature flag for grab (not needed — always installed) -- Changes to `pos media mp3` or `pos media mp4` (consumed as-is) - -[DECIDED] - ---- - -## Decision 2: URL Classification Logic - -### Problem - -How does `pos media grab` determine whether a URL should be downloaded as audio (mp3) or video (mp4)? - -### Evidence - -- `music.youtube.com` URLs are always audio-only (music streaming). -- YouTube regular/shorts URLs are primarily video content. -- SoundCloud, Bandcamp are audio-first platforms. -- yt-dlp handles both audio and video for all supported sites. -- The user's primary use case: YouTube music → ~/Music, YouTube video → ~/Videos. - -### Classification Rules (Priority Order) - -| Pattern | Classification | Reasoning | -|---------|---------------|-----------| -| `*music.youtube.com*` | `audio` | YouTube Music is audio-only streaming | -| `*soundcloud.com*` | `audio` | Audio-first platform | -| `*bandcamp.com*` | `audio` | Audio-first platform | -| `*youtube.com*`, `*youtu.be*` | `video` | YouTube primary: video content | -| `*youtube.com/shorts/*` | `video` | Short-form video | -| `*vimeo.com*` | `video` | Video platform | -| `*twitch.tv*` | `video` | Video streaming | -| Everything else | `video` (default) | Safe default — yt-dlp handles format negotiation | - -### Decision - -Domain-based regex classification with a configurable default. - -### Implementation - -```bash -classify_url() { - local url="$1" mode="${GRAB_DEFAULT:-video}" - case "$url" in - *music.youtube.com*) echo "audio" ;; - *soundcloud.com*) echo "audio" ;; - *bandcamp.com*) echo "audio" ;; - *youtube.com*|*youtu.be*) echo "video" ;; - *vimeo.com*) echo "video" ;; - *twitch.tv*) echo "video" ;; - *) echo "$mode" ;; - esac -} -``` - -### Why Not yt-dlp `--dump-json`? - -Using `yt-dlp --dump-json` to detect content type (e.g., checking for audio-only formats) would: -- Require a network round-trip per URL (slow — 2-5s on YouTube) -- Fail on age-gated content without cookies -- Add unnecessary complexity for a heuristic - -Domain-based classification is instant, reliable for the primary use case, and covers 95%+ of real URLs. The `--audio`/`--video` flag overrides for edge cases. - -### Overrides - -- `--audio` forces mp3 regardless of classification -- `--video` forces mp4 regardless of classification -- `--best` is passed to mp4 by default (non-interactive mode — see Decision 4) - -[DECIDED] - ---- - -## Decision 3: Config Scope - -### Problem - -Does `pos media grab` need its own config scope? What settings? - -### Evidence - -- `pos media mp3` has `OUT_DIR=$HOME/Music` -- `pos media mp4` has `OUT_DIR=$HOME/Videos` -- Both accept `--output` flag override -- The user might want all grabs to go to a single directory -- The user might want a different default mode (e.g., always audio for YouTube Music) - -### Decision - -Config scope `grab` with one key: - -```bash -# POS_CONFIG: grab | grab.env | GRAB_DEFAULT=:Default mode for unknown domains (video or audio, default video) -``` - -### Rationale - -Minimal config — the tool's primary job is routing, not download settings. The `GRAB_DEFAULT` key lets the user change the fallback for unrecognized domains without code changes. Output directories are inherited from mp3/mp4 and overridable via `--output`. - -### Future Extensibility - -If needed, future keys could include: -- `GRAB_MUSIC_DIR` (override mp3 output dir) -- `GRAB_VIDEO_DIR` (override mp4 output dir) -- `GRAB_COOKIES` (shared cookies file for all grabs) - -Not implemented now — premature without user demand. - -[DECIDED] - ---- - -## Decision 4: Non-Interactive Default for Telegram Context - -### Problem - -`pos media mp4` has an interactive format selector (prompts on TTY). When called from the Telegram listener via `run_and_reply` (which uses `bash -c "$cmdline"` with no TTY), the prompt would hang or be swallowed by `tee`. - -### Evidence - -- `pos media mp4` lines 92-116: interactive `read -rp "Format ID"` when no `--format`/`--best`/`--worst` is specified -- `pos` dispatcher line 287: interactive tools are in `INTERACTIVE_CMDS` which skips `tee` logging -- The listener's `run_and_reply` (line 349) runs commands via `timeout "$tmo" bash -c "$cmdline"` — no TTY -- `spawn` helper (lib/common.sh line 76) runs commands in background — no TTY interaction - -### Decision - -`pos media grab` passes `--best` to `pos media mp4` by default. Users can override with `--worst` flag on grab. - -### Implementation - -```bash -# When routing to mp4, always add --best unless user specified --worst -MP4_ARGS=(--best) -[ "$WORST" -eq 1 ] && MP4_ARGS=(--worst) -``` - -### Why --best and Not --worst? - -- Best quality is the expected default when sending a video link to a bot -- "I want to download this video" implies "I want it to look good" -- The `--worst` flag exists for bandwidth-constrained scenarios (explicit opt-in) - -### Interaction with pos media mp4 - -`pos media mp4` already supports `--best` and `--worst` flags (line 4, 87-88). No changes needed to mp4. - -[DECIDED] - ---- - -## Decision 5: Listener Integration — URL Routing Step - -### Problem - -Where does URL detection go in the listener's routing chain, and how does it work? - -### Evidence - -Current `handle_message` routing (bin/pos-communication-telegram-listener lines 658-731): - -``` -/help|/start → reply mapped commands list - ↓ (no match) -text-prefix map → run mapped command with text as argument - ↓ (no match) -AI bridge → forward to Gemini - ↓ (no match) -command map → run /command - ↓ (no match) -"Unknown command" -``` - -The user sends bare URLs from their phone. These should be detected and routed to `pos media grab`. - -### Decision - -New routing step between prefix map and AI bridge: - -``` -/help|/start → reply mapped commands list - ↓ (no match) -text-prefix map → run mapped command - ↓ (no match) -**URL detect → pos media grab** (NEW) - ↓ (no match) -AI bridge → forward to Gemini - ↓ (no match) -command map → run /command - ↓ (no match) -"Unknown command" -``` - -### Why After Prefix Map? - -- A prefixed command like `ai https://...` should go to the AI bridge, not grab -- A bare URL with no prefix should go to grab -- The prefix map is explicit user configuration — it takes priority - -### Why Before AI Bridge? - -- A bare URL has no AI intent — it's a download request -- The AI bridge would waste time (and API credits) analyzing a URL -- Future non-AI intents (reminders, etc.) slot in as more case arms here - -### Implementation - -Add to `handle_message` after the prefix map block (after line 683, before line 685): - -```bash - # URL detect: bare HTTP(S) URLs → pos media grab - local grab_url - if grab_url="$(url_detect "$text")"; then - log "grab: $grab_url" - run_and_reply "pos media grab --best \"$grab_url\"" "$msg_id" 600 - return - fi -``` - -Add `url_detect` function before `handle_message`: - -```bash -# Detect a bare URL in message text. Extracts the first http(s) URL. -# Returns 0 + prints the URL on success, 1 if no URL found. -url_detect() { - local text="$1" - local url="" - # Match http:// or https:// followed by non-whitespace - if [[ "$text" =~ (https?://[^[:space:]]+) ]]; then - url="${BASH_REMATCH[1]}" - # Strip trailing punctuation that's likely not part of the URL - url="${url%%[,.\)!?:;]}" - url="${url%%\>*}" - [ -n "$url" ] || return 1 - printf '%s' "$url" - return 0 - fi - return 1 -} -``` - -### Edge Cases - -| Input | `url_detect` result | Routing | -|-------|-------------------|---------| -| `https://youtube.com/watch?v=xyz` | `https://youtube.com/watch?v=xyz` | grab → mp4 | -| `check out https://youtu.be/xyz` | `https://youtu.be/xyz` | grab → mp4 | -| `ai what is https://example.com` | `https://example.com` | grab (not AI!) | -| `/status` | (no match) | command map | -| `opencode check cpu` | (no match) | prefix map | -| `hello world` | (no match) | AI bridge or unknown | - -The `ai what is https://...` case is a minor trade-off — the user is more likely asking AI about the URL content than wanting to download it. However, this is a rare edge case, and the primary use case (bare URL from phone) is served correctly. If it becomes an issue, the URL detection could be refined to only trigger when the URL is the dominant content (e.g., text length < 2x URL length). - -### Timeout - -600 seconds (10 minutes). Reasonable for most videos. The listener's `run_and_reply` already handles timeout gracefully (returns "exit N" + output). - -[DECIDED] - ---- - -## Decision 6: Tool Output Contract - -### Problem - -What does `pos media grab` print so the listener can reply to the user? - -### Evidence - -- `pos media mp3` line 85: `spawn "downloading audio → $OUT_DIR"` — prints via spawn (OK/FAIL + elapsed) -- `pos media mp4` line 131: `spawn "downloading video → $OUT_DIR"` — same -- The listener's `run_and_reply` (line 349-360): captures stdout+stderr, replies with output (truncated to 3800 chars) -- Other tools: `pos system health` prints multi-line reports that the listener forwards verbatim - -### Decision - -`pos media grab` prints clean, user-friendly output to stdout: - -``` -🎵 Downloaded: Artist - Title (3:42) -📁 ~/Music/Artist - Title.mp3 (4.2 MB) -``` - -or - -``` -🎬 Downloaded: Video Title (10:15) -📁 ~/Videos/Video Title.mp4 (125 MB) -``` - -Error case: - -``` -❌ Download failed: [yt-dlp error summary] -``` - -### Implementation - -After the mp3/mp4 delegation succeeds, the tool: - -1. Runs `yt-dlp --print title --print duration_string --print filesize_approx "$URL"` to fetch metadata (fast, no download) -2. Runs `stat --printf='%s' "$filepath"` to get actual file size -3. Prints the formatted summary - -For errors: capture stderr from the delegated command, print a clean error line. - -### Why Not Just Forward spawn Output? - -The `spawn` helper prints spinner text + OK/FAIL, which is good for terminal but not for Telegram. A structured summary (title, path, size) is more useful when you get a Telegram notification about a download. - -### Why Metadata is a Separate Call? - -The download itself (via mp3/mp4) doesn't expose title/size in a parseable format. The metadata call is fast (~1s) and gives us the info we need for the summary. - -[DECIDED] - ---- - -## Decision 7: Testing Strategy - -### Problem - -How to test without a real Telegram bot or yt-dlp network access? - -### Evidence - -- DEV.md "Testing tools that need root / systemd / missing deps" (lines 194-214) -- Existing patterns: `FLAGS_DIR`, `SMB_CONF` env-seam approach; stub PATH fakes -- The tool is a thin classifier + delegator — most logic is in URL matching - -### Decision - -Stub PATH approach with fake `pos media mp3`/`pos media mp4` scripts. - -### Test Harness Layout - -``` -/tmp/opencode/media-grab-test/ -├── run-tests.sh # Main test runner -├── stubs/ # Fake binaries -│ ├── pos # Fake pos dispatcher → delegates to stub mp3/mp4 -│ ├── pos-media-mp3 # Echoes args, creates a fake file -│ ├── pos-media-mp4 # Echoes args, creates a fake file -│ ├── yt-dlp # Echoes args, creates a fake file, prints metadata -│ ├── stat # Returns fake file size -│ └── ffmpeg # No-op -└── fixtures/ # Test URLs (various domains) -``` - -### Test Cases (20+) - -| # | Test | Input | Expected | -|---|------|-------|----------| -| 1 | YouTube Music URL | `https://music.youtube.com/watch?v=xyz` | Routes to mp3 | -| 2 | YouTube video URL | `https://youtube.com/watch?v=xyz` | Routes to mp4 --best | -| 3 | YouTube short URL | `https://youtu.be/xyz` | Routes to mp4 --best | -| 4 | YouTube shorts URL | `https://youtube.com/shorts/xyz` | Routes to mp4 --best | -| 5 | SoundCloud URL | `https://soundcloud.com/artist/track` | Routes to mp3 | -| 6 | Bandcamp URL | `https://bandcamp.com/album/track` | Routes to mp3 | -| 7 | Vimeo URL | `https://vimeo.com/123456` | Routes to mp4 --best | -| 8 | Unknown domain | `https://example.com/video.mp4` | Routes to mp4 --best (default) | -| 9 | `--audio` override | `--audio https://youtube.com/watch?v=xyz` | Routes to mp3 | -| 10 | `--video` override | `--video https://soundcloud.com/track` | Routes to mp4 | -| 11 | `--worst` flag | `--worst https://youtube.com/watch?v=xyz` | Routes to mp4 --worst | -| 12 | `--best` explicit | `--best https://youtube.com/watch?v=xyz` | Routes to mp4 --best | -| 13 | `--dry-run` | `--dry-run https://youtube.com/watch?v=xyz` | Prints command, no download | -| 14 | No URL | (empty) | usage | -| 15 | `--help` | `--help` | Shows usage | -| 16 | Invalid URL | `not-a-url` | Error: "not a valid URL" | -| 17 | GRAB_DEFAULT=audio | `GRAB_DEFAULT=audio https://unknown.com/x` | Routes to mp3 | -| 18 | `--output` override | `--output /tmp/test https://...` | Passes to mp3/mp4 | -| 19 | HTTP URL | `http://youtube.com/watch?v=xyz` | Routes to mp4 (http not https) | -| 20 | `--no-playlist` | `--no-playlist https://youtube.com/playlist?list=xyz` | Passes to mp3/mp4 | - -### Stub Implementation - -```bash -#!/usr/bin/env bash -# Fake pos-media-mp3 — records args, creates a dummy file -OUT_DIR="${OUT_DIR:-/tmp/test-output}" -mkdir -p "$OUT_DIR" -URL="${*: -1}" # last arg is URL -echo "pos-media-mp3 called with: $*" > /tmp/test-output/mp3.log -touch "$OUT_DIR/test.mp3" -echo "Downloaded: Test Song (3:42)" -echo "📁 $OUT_DIR/test.mp3 (4.2 MB)" -``` - -### What NOT to Test - -- yt-dlp actual behavior (covered by yt-dlp's own tests) -- The listener integration (tested manually or with a Telegram bot test harness) -- URL regex edge cases that are handled by bash `[[ =~ ]]` (well-tested in bash) - -[DECIDED] - ---- - -## Decision 8: Dependency Handling - -### Problem - -What deps does `pos media grab` declare, and where? - -### Evidence - -- `pos media mp3` lines 17-18: guards `yt-dlp` and `ffmpeg` -- `pos media mp4` lines 17-18: same guards -- `pos media grab` delegates to mp3/mp4, which handle their own deps -- Grab itself only needs bash builtins for URL classification - -### Decision - -No deps guard in `pos media grab`. Deps are the responsibility of mp3/mp4. - -### Rationale - -Grab is a pure classifier + delegator. It doesn't call yt-dlp directly. If mp3/mp4 are called and their deps are missing, they'll error with their own helpful messages. Adding redundant deps guards in grab would: -- Duplicate error messages -- Create maintenance overhead when mp3/mp4 deps change -- Violate single-responsibility (grab classifies, mp3/mp4 download) - -### Exception - -If grab needs `stat` for file size reporting (Decision 6), that's a standard coreutil — no guard needed on Debian/Ubuntu. - -[DECIDED] - ---- - -## Summary: Implementation Scope - -### Files to Create - -| File | Responsibility | -|------|---------------| -| `bin/pos-media-grab` | URL classifier, delegate to mp3/mp4, clean output | - -### Files to Modify - -| File | Change | -|------|--------| -| `bin/pos-communication-telegram-listener` | Add `url_detect` function + URL routing step in `handle_message` | - -### Files to Update (Docs) - -| File | Change | -|------|--------| -| `DOC/POS.md` | Add `pos media grab` row to media table + detail block | -| `DOC/HOWTO.md` | Add media section reference if not already present | -| `DOC/howto/media.md` | Add `grab` usage example | -| `DOC/AGENT_Context_Project.md` | (auto via `make gen`) tree, dispatch, filetable | -| `AGENT_TODO.md` | Move to Done | - -### Post-Implementation Gates - -```bash -chmod +x bin/pos-media-grab -bash -n bin/pos-media-grab -bash -n bin/pos-communication-telegram-listener -make gen && make check && make lint # 0 FAIL, 0 WARN -``` - ---- - -## Handoff - -Status: DECISION_READY - -Problem: Telegram bot needs intelligent URL routing to auto-download content as audio/video. - -Decision: New `bin/pos-media-grab` tool (domain classifier + mp3/mp4 delegator) + listener URL routing step between prefix map and AI bridge. - -Reasoning: Evidence-based — domain classification is fast/reliable for the primary use case, existing mp3/mp4 tools handle the heavy lifting, listener routing chain slots naturally for new intents. - -Ownership: `bin/pos-media-grab` (new tool), listener integration in existing listener. - -Interfaces: grab is called by the listener; grab calls mp3/mp4. - -Approved scope: `bin/pos-media-grab` (new), listener modification (add ~30 lines), doc updates. No changes to mp3/mp4. - -Explicitly out of scope: Changes to mp3/mp4 tools, feature flags, complex config. - -Constraints: Must pass `make gen && make check && make lint` (0 FAIL, 0 WARN). Follow all pos conventions (shebang, strict mode, `# POS:` header, deps-before-help pattern, `common.sh` sourcing). - -Verification: Stub-PATH test suite (20+ cases), `bash -n` on both modified files, `make gen && make check && make lint`. - -Risks: -- URL regex may not match all URL formats (mitigated: covers http/https, the dominant formats) -- `--best` may not be what the user always wants for video (mitigated: `--worst` override) -- Metadata fetch adds ~1s latency (mitigated: acceptable for download context) - -Recommended next agent: Builder - -Reason: Architecture is fully defined with specific file paths, function signatures, listener line changes, config keys, and test plan. Ready for implementation. - -Architect changes: None — pure decision record. diff --git a/AgentsReport/architect/2026-09-06_ai-cost-window-design.md b/AgentsReport/architect/2026-09-06_ai-cost-window-design.md deleted file mode 100644 index da4efaf..0000000 --- a/AgentsReport/architect/2026-09-06_ai-cost-window-design.md +++ /dev/null @@ -1,307 +0,0 @@ -# AI cost & session-window design — OpenRouter 402 + bounded session memory - -**Date:** 2026-09-06 -**Author:** Architect (big-pickle) -**HEAD:** 8ce5479 (clean tree) -**Mode:** design-only (no file edits besides this report) - -## TL;DR - -Two tightly associated defects in `pos ai` / `bin/pos-ai`: - -1. **402 root cause** — no provider ever sends `max_tokens`, so OpenRouter's credit - pre-check charges the routed model's full worst-case output (131072 on - `openrouter/auto`) and rejects balances under that. Fix: send an explicit capped - `max_tokens`. -2. **Unbounded-ish session** — `MAX_SESSION_TURNS=40` counts *messages* (20 - exchanges); user wants "last 5 req/response" = 10 messages. - -**Decisions:** add `AI_MAX_TOKENS` (num, default **2048**) honored by openrouter -(body `max_tokens`) and gemini (`generationConfig.maxOutputTokens`); **skip** llamacpp. -Add `AI_SESSION_TURNS` (num, default **40** — backward compatible) honored lazily in -`session_push`; user sets **10** for 5 pairs. Both vars declared in the `@General` -section of the `# POS_CONFIG:` header, docs updated. No chat/alias wrapper changes. - -**Open items:** none blocking. Tester should add provider-body + session-pruning -coverage (see §Testing). - ---- - -## Verified fact confirmation (A–D) - -All user-reported facts confirmed against source at HEAD: - -**A — 402 root cause: CONFIRMED.** -- `lib/ai-providers/openrouter.sh:22-23` — body is only `{model,messages}`; no - `max_tokens`. `:24-29` POSTs straight to OpenRouter with unchanged body. -- `lib/ai-providers/gemini.sh:17-19` — body only `{contents,...}`; no - `generationConfig`. (`:20-23` adds only `systemInstruction`.) -- `lib/ai-providers/llamacpp.sh:32-33` — body `{model,messages,stream:false}`. -- Conclusion: none carry a generation cap → OpenRouter 402 with the user's thin - balance. Fix is to send an explicit `max_tokens`. - -**B — session window: CONFIRMED.** -- `bin/pos-ai:25` `MAX_SESSION_TURNS=40`. -- `session_push()` `bin/pos-ai:285-290` → `'.messages |= .[-"$MAX_SESSION_TURNS":]'` - prunes to last N **messages** (40 msgs = 20 exchanges). User wants 5 pairs = 10 messages. - -**C — config surface: CONFIRMED.** -- `# POS_CONFIG:` header `bin/pos-ai:6`, scope `ai | ai.env`. -- `num:` type already used in the same header (`LLAMACPP_CTX_SIZE=num:…`, - `LLAMACPP_GPU_LAYERS=num:…`). -- Validation: `lib/config-ui.sh:419` `*,num,*) [[ "$val" =~ ^-?[0-9]+$ ]]` — integer-only - on *entry*; empty input ="kept current value" (`:468-471`); `-` = clear (`:472-477`). - So `num:` + empty/unset → falls back to code default. Clean. -- Env precedence: `load_env_file` (`lib/config-ui.sh:336-357`) exports a file key only - when the variable is not already set in the environment (`:351-353`) → env beats - file beats default. Providers read config via env (`AI_API_KEY` pattern). -- `PROVIDER_CONFIG` headers exist on `openrouter.sh:7-8`, `gemini.sh:7-8`, - `llamacpp.sh:7` for *provider-specific* keys. - -**D — provider resolution: CONFIRMED.** -- `bin/pos-ai:679-683` — `--provider` flag > `AI_PROVIDER` env > `gemini`. -- `load_config()` `bin/pos-ai:141-151` loads `ai.env` (+ legacy files); called by - `resolve_key` (line 155) and at provider resolution (line 681). -- Call site `bin/pos-ai:522` (ask) and `:557` (chat): `provider_generate "$model" "$messages" "$system"`. - -**Gates/tests scan:** -- No test pins `MAX_SESSION_TURNS`, the provider request bodies, or header text - (`t-config-precedence.sh` targets `pos-ai-server`, a separate tool). No forced - test update. -- `DOC/POS.md` has a hand-maintained ai.env config table (`:88-97`) and mentions - "capped at 40 turns" at `:64`. `DOC/HOWTO.md:45` and `DOC/AGENT_Context_Project.md:491` - list ai.env vars (hand-maintained). All need doc rows/bumps for the new vars. - ---- - -## Decision 1: `AI_MAX_TOKENS` — cap generation tokens - -**Status: [DECIDED]** - -### Options & trade-offs - -**Option 1 (chosen) — single global `AI_MAX_TOKENS=num`, default 2048, honored by remote providers (openrouter + gemini); skip llamacpp.** -- *Advantages:* smallest change that fixes the 402 (OpenRouter pre-check sees a - capped cost) and is a real per-request cost ceiling; one var, one default; fits - the existing `AI_*` env naming and the tool-level `@General` config section; no - new per-provider surface. -- *Costs:* remote providers share one ceiling (no per-provider cap without user - intervention). -- *Risks:* a too-low cap truncates long answers — mitigated by default 2048 being - ample for terse `ask`/`chat` CLI answers; user can raise it. -- *Reasoning for default 2048:* conservative (user's balance affords ~4511 tokens at - routed price, so 2048 passes the pre-check with margin) while being a practical, - real ceiling. 2048 tokens ≈ several thousand chars — plenty for the terse, - commands-first assistant role this tool plays. - -**Option 2 — per-provider caps via `PROVIDER_CONFIG` (e.g. `AI_MAX_TOKENS` on openrouter.sh, gemini.sh).** -- *Advantages:* independent ceilings per provider. -- *Costs:* two declarations, redundant section plumbing, and the *default* (which is - the entire point) still has no shared home → awkward. `PROVIDER_CONFIG` is for - provider-specific concerns; a cost ceiling + 402 pre-check over both remote - providers is tool-level, not provider-specific. - -**Option 3 — no gemini cap; only openrouter.** -- *Advantages:* minimal (402 only affects OpenRouter). -- *Costs:* leaves Gemini without any cost ceiling while introducing the same var — - inconsistent, and Gemini's own pricing can surprise. Rejected. - -**Option 4 — include llamacpp too (`max_tokens` in body; it accepts it).** -- *Advantages:* provider parity on the OpenAI-compatible endpoint. -- *Costs:* local & free — no credit pre-check, no cost. Adds surface with zero user - benefit. Rejected on the "smallest sufficient design" principle. - -### Implementation contract -- Env var name: **`AI_MAX_TOKENS`**, type `num`, default **2048**. -- OpenRouter body (`openrouter.sh:22-23`): add `max_tokens`. -- Gemini body (`gemini.sh:17-19`): add `generationConfig.maxOutputTokens`. -- llamacpp: **no change**. -- Providers read `"${AI_MAX_TOKENS:-2048}"` from env; value is present because - `load_config` runs before `provider_generate` (resolve path confirmed in D). -- Declare in `@General` section of `# POS_CONFIG:` header (`bin/pos-ai:6`). - ---- - -## Decision 2: `AI_SESSION_TURNS` — bounded session window - -**Status: [DECIDED]** - -### Options & trade-offs - -**Option 1 (chosen) — new `AI_SESSION_TURNS=num`, default 40 (unchanged), resolved lazily in `session_push`.** -- *Advantages:* fully backward compatible — no silent memory truncation for existing - users. The user's "last 5 req/response" = setting `AI_SESSION_TURNS=10`. One var, - one default. -- *Costs:* existing users must opt in (they already have the 40 behavior, so no - regression). -- *Why keep default 40:* backward compatibility is a hard project value; silently - shifting the default changes session context for *every* user, discards history - they may rely on, and is a behavioral change not requested globally (only for this - user). Keep 40. - -**Option 2 — change the default to 10.** -- *Advantages:* meets the stated want out-of-the-box. -- *Costs:* silent behavior change for all users; discards memory; not requested - globally. Rejected — keep the change opt-in via the new var. - -### Semantics (must be documented) -The existing prune is `.messages |= .[-N:]` where N counts **messages** — 2 messages -per exchange. So `AI_SESSION_TURNS=10` ⇒ last **5** exchanges (5 user + 5 assistant). -The config description must state: "message count (2 per exchange); 10 = last 5 -exchanges". - -### Implementation constraint — lazy resolution (important) -`MAX_SESSION_TURNS` is currently assigned at `bin/pos-ai:25`, which executes at -top-level **before** `load_config` is first called (line 681). If we wrote -`MAX_SESSION_TURNS="${AI_SESSION_TURNS:-40}"` at line 25, an `AI_SESSION_TURNS` set -*only in ai.env* would not yet be loaded → always 40. - -Therefore: -- Keep **line 25** as-is (`MAX_SESSION_TURNS=40`), used for the help text (`:67` - shows the default, accurate). -- In `session_push()` (lines 285-290), resolve **lazily**: - `local n="${AI_SESSION_TURNS:-$MAX_SESSION_TURNS}"` and use `$n` in the jq prune. - Because `session_push` runs inside `cmd_ask`/`cmd_chat` — after `load_config` - (via `resolve_key` at `:499`/`:538`) has exported `AI_SESSION_TURNS` into the - process env — the ai.env value is honored. Shell-exported `AI_SESSION_TURNS` - wins too (env-wins in `load_env_file`). -- Help text `bin/pos-ai:67` stays accurate ("capped at $MAX_SESSION_TURNS turns") - since the default remains 40. Optionally add a `Config:` help line documenting the - var — recommended, and it confirms "turns = messages, 10 = 5 pairs". - ---- - -## Decision 3: `# POS_CONFIG:` header change - -**Status: [DECIDED]** - -- Single `@General` section addition (not per-provider `PROVIDER_CONFIG`): - - `AI_MAX_TOKENS=num:Max output tokens per request (default 2048; OpenRouter/Gemini cost cap)` - - `AI_SESSION_TURNS=num:Session message cap — 2 per exchange (default 40 = 20 exchanges; 10 = last 5)` -- Place both in the `@General` section alongside `AI_SYSTEM_PROMPT` (end of the - long header line, `bin/pos-ai:6`). -- **Why @General, not PROVIDER_CONFIG:** the default is shared across providers - (2048, 40) and both caps are tool-level concerns. `PROVIDER_CONFIG` is reserved for - provider-specific keys (API keys, models). A single global declaration is the - cleanest and avoids duplicating the default in two provider files. -- `num:` type confirmed suitable: `lib/config-ui.sh:419` enforces integer on entry; - empty=keep current, `-`=clear (`:468-477`); unset → code default. No empty-parse - concern. -- Provider bodies consume the vars from env, so no `PROVIDER_CONFIG` additions are - needed on `openrouter.sh`/`gemini.sh`. (They could be added later if per-provider - caps are ever wanted — out of scope now.) - ---- - -## Decision 4: Docs & gates - -**Status: [DECIDED]** - -- **`DOC/POS.md`** (hand-maintained): - - ai.env table (`:88-97`): add rows for `AI_MAX_TOKENS` (no/`2048`/"Max output - tokens per request (OpenRouter/Gemini cost cap)") and `AI_SESSION_TURNS` - (no/`40`/"Session message cap — 2 per exchange; 10 = last 5 exchanges"). The - `AI_SYSTEM_PROMPT` row (`:93`) is the placement anchor. - - Line 64 text "capped at 40 turns" remains true (default unchanged) — no edit - strictly needed, but a short parenthetical "(configurable via AI_SESSION_TURNS)" - is recommended. -- **`DOC/HOWTO.md:45`** — append `AI_MAX_TOKENS`, `AI_SESSION_TURNS` to the listed - ai.env vars. -- **`DOC/AGENT_Context_Project.md:491`** — append the two vars to the ai.env - summary parenthetical (hand-maintained). -- **`make gen`**: header text change does not add commands/subcommands/flags, so the - generated tree/dispatch tables are unaffected; `completions/pos.bash` config-scope - table regenerates to include the new keys. Run `make gen` (deterministic, `LC_ALL=C` - per convention), then `make check`, then `make lint`. -- **Line-count rows** above the filetable marker in `AGENT_Context_Project.md`: only - bump a row if a `pos-*`/`lib/*` file's length changes (it does — lib/ai-providers - grow; bin/pos-ai grows). Do not touch rows for files that don't change. -- **tests**: existing suite does not pin the session default or provider bodies, so - nothing is *forced*. Recommend new coverage (§Testing). - ---- - -## Decision 5: Scope fence - -**Status: [DECIDED]** - -**Approved outcome:** OpenRouter 402 eliminated (explicit capped `max_tokens` on -remote providers) and session history bounded via configurable `AI_SESSION_TURNS`. - -**In-scope files:** -- `bin/pos-ai` — `# POS_CONFIG:` header (line 6, @General additions); `session_push` - lazy `AI_SESSION_TURNS` resolution (lines 285-290); optional `Config:` help lines - for the two vars. Line 25 stays `MAX_SESSION_TURNS=40`. -- `lib/ai-providers/openrouter.sh` — add `max_tokens` to body (lines 22-23). -- `lib/ai-providers/gemini.sh` — add `generationConfig.maxOutputTokens` (lines 17-19). -- `lib/ai-providers/llamacpp.sh` — **no change**. -- Docs (hand-maintained): `DOC/POS.md`, `DOC/HOWTO.md`, `DOC/AGENT_Context_Project.md`. -- tests (later, Tester). - -**Allowed interface changes:** two new config keys in scope `ai`; provider request -bodies gain a token cap. Provider call signature `provider_generate "$model" "$messages" "$system"` -is unchanged. - -**Explicitly out of scope:** `bin/pos-ai-server`, chat/alias wrappers, `pos-ai-alias`, -HuggingFace downloader, other tools, per-provider caps, changing `MAX_SESSION_TURNS` -default, any llmacpp body change. - -**Architectural constraints:** -- No new provider-side config plumbing; vars read from env (`AI_*` convention). -- Backward compatible: session default 40 and empty/unset values fall back to - documented defaults. -- Deterministic `make gen`; `make check` + `make lint` green (definition of done). - -**Open risks:** -- `AI_MAX_TOKENS` too low truncates long `--full` answers — default 2048 mitigates; - user can raise. -- env-vs-file precedence: exported env var beats config file (per `load_env_file`) — - expected and documented. - -**Verification:** -- `bash -n` clean; `make gen`/`make check`/`make lint` green. -- A 402 reproduction no longer triggers when `AI_MAX_TOKENS` is set/at default. -- `AI_SESSION_TURNS=10` prunes the session to last 5 exchanges. - ---- - -## Testing budget (suggestion for Tester / Builder verification) - -- **Provider body cap:** source each provider adapter in a sandbox with `curl` - stubbed, assert the request JSON contains `max_tokens` (openrouter) / a truthy - `generationConfig.maxOutputTokens` (gemini). Pattern: `t-config-precedence.sh` style - curl-log capture with a fake `curl` in `PATH` (see existing tests). -- **Session pruning:** drive `session_push` with a crafted messages JSON and - `AI_SESSION_TURNS=10`, assert exactly the last 10 messages remain (5 pairs); and - `AI_SESSION_TURNS` unset → 40 retained (default commits to backward compat). -- **Help/default fidelity:** `pos ai --help` still shows 40; `pos config ai` lists the - two new `num:` keys and rejects a non-integer (`cfg_validate`). -- **Regression:** existing `t-config-precedence.sh`, `run-tests` full suite green. - ---- - -## Handoff - -**Status:** DECISION_READY - -**Problem:** OpenRouter 402 (no `max_tokens` → full worst-case pre-check) and -unbounded session memory; user wants 5 req/response. - -**Decision:** add `AI_MAX_TOKENS=num` (default 2048) honored by openrouter + gemini -(skip llamacpp); add `AI_SESSION_TURNS=num` (default 40, backward compatible) resolved -lazily in `session_push`; both in `@General` config section; docs updated. - -**Ownership:** `bin/pos-ai` + `lib/ai-providers/{openrouter,gemini}.sh` (+ docs). - -**Interfaces:** two new `ai`-scope config keys; provider bodies gain a token cap. -Call signature unchanged. - -**Approved scope / constraints / verification / out-of-scope:** see Decision 5. - -**Risks:** see Decision 5 (token-cap truncation; env-vs-file precedence — all mitigated). - -**Recommended next agent:** **Builder** - -**Reason:** the architecture and scope are fully specified with exact line-level -changes (no architectural ambiguity left). Builder can implement without making -architecture decisions. Tester follows for the recommended coverage. diff --git a/AgentsReport/architect/2026-09-06_ai-server-fix-design.md b/AgentsReport/architect/2026-09-06_ai-server-fix-design.md deleted file mode 100644 index b72a69c..0000000 --- a/AgentsReport/architect/2026-09-06_ai-server-fix-design.md +++ /dev/null @@ -1,177 +0,0 @@ -# AI Server Start Breakage — Root Cause + Fix Design Decisions - -**Date:** 2026-09-06 -**Architect:** Design pass over the Detective report `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (root cause classified **FACT**). -**Constraint:** No code changes in this pass; decisions + ratified fix scope (F1-F7) for the Builder; test-fixture requirements for the Tester. Preserve the existing Bash tool architecture and the D-A..D-F stabilization decisions. - ---- - -## TL;DR - -1. **DQ1 — Validation stays help-gated.** The SIGPIPE race (F2) is the only validation defect; help-based validation is binary-reality-based and correct once deterministic. Version is cosmetic. "unknown" means: validate against real `--help` regardless, message without the version. NO version→capability map. [DECIDED] -2. **DQ2 — `resolve_model` gains file-inside-dir expansion.** `$HF_DOWNLOAD_DIR/`/dir → exactly-one `*.gguf` resolves to it; multiple → list + err ("pick one"); zero → err as today. Also accept `/.gguf`. Precedence: absolute path > dir-with-exactly-one-gguf > `$HF_DOWNLOAD_DIR/.gguf` flat file > `$HF_DOWNLOAD_DIR//` > relative-as-is. Existing bare-file behavior unchanged (backward compatible). [DECIDED] -3. **DQ3 — User-bus pre-flight by default, plus `--no-unit` direct-run escape hatch (option c).** Pre-flight aborts BEFORE writing the unit (no orphan); `--no-unit` runs the server directly under nohup+pidfile for headless/SSH boxes. Shared helper `ensure_user_bus` lives in `lib/common.sh` (all three tools source it). Matrix-listener + network-download get the same pre-flight. Remediation text includes `export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`. [DECIDED] -4. **DQ4 — `find_llamacpp` drops bare `server` and `llama.cpp/server`.** Keep `llama-server`, `llama-server-cuda`. Do NOT add `llama-server-mtl` (Mac-only; out of target). Stale-unit: start F4 pre-flight WARNS about an existing unit, does not delete it (never gratuitously remove user state). [DECIDED] -5. **DQ5 — Always pin `--port $PORT`.** Once F2 makes validation reliable, ExecStart always carries `--port $PORT` (help-gated, so a build without `--port` omits it). No adapter probing fallback change needed (deferred). [DECIDED] -6. **DQ6 — Installer sanity IS in scope.** `apps/ai/llamacpp.sh` post-install runs `llama-server --version` (2>&1) + `--help` and errs if the binary doesn't execute; verifies the symlink target exists. [DECIDED] - ---- - -## DQ1: Validation posture when version is unknown - -### Decision -**Keep help-based validation as the single source of truth. Do NOT add a version→capability map.** "unknown" means: still validate against the real `--help` output; only the version string in log/error messages is dropped (message proceeds without version). - -### Rationale -Help-based validation reflects what the actual installed binary supports — it is always correct once the SIGPIPE race (F2) is fixed, because it peers at real capability output. Adding a version→capability table would introduce a second, version-coupled source of truth that drifts the moment llama.cpp adds/renames flags; its only benefit would be to decorate messages with a version string, which is cosmetic. The version is useful for *debugging* ("which build am I running") but must never gate validation. "unknown" therefore degrades only to "no version in messages", not to "no validation". - -### How "unknown" behaves -- `detect_llama_version` returns "unknown" only when `--version` is genuinely unreadable (no binary) — after F1, stderr is captured and the output IS readable for real builds. -- With "unknown", `validate_*_flags` still fetch `--help` and judge flags normally; error texts simply omit the version token. -- F1's broadened regex handles both semver and build-only strings, so "unknown" becomes rare (only truly missing binary/--version). - -[DELIVERY: F1 capture-2>&1 + broadened regex; F2 race fix. No capability map.] - ---- - -## DQ2: `resolve_model` directory expansion - -### Decision -Extend `resolve_model` (`bin/pos-ai-server:230-264`) to resolve a directory-typed explicit argument when it contains a usable model file, with exact precedence. Interactive `pick_model` stays recursive (already lists flat `.gguf` files one level up via `find -type f`; confirmed at `:210` — it lists each dir's files, so no F3 change needed there). - -### Precedence (highest → lowest) -1. **Absolute path** that is a file (`[ -f "$explicit" ]`) — unchanged; if the absolute path is a directory, treat as directory case below. -2. **`$HF_DOWNLOAD_DIR/` is a DIRECTORY** containing exactly one `*.gguf` → resolve to that file. If multiple `*.gguf` → print all and `err "pick one: /"`. If zero → fall through / err (never silently pick). -3. **`$HF_DOWNLOAD_DIR/.gguf` flat file** — unchanged (`[ -f "$candidate" ]`). -4. **`$HF_DOWNLOAD_DIR//.gguf`** (slug/file form) — resolve to `$HF_DOWNLOAD_DIR//.gguf`. -5. **Relative-as-is** — unchanged (`[ -f "$explicit" ]`). - -### Backward compatibility -Bare existing behavior (flat file path, absolute file path, relative path) is fully preserved — the new directory-expansion branches only fire where the current code would have errored with "Model not found". No silent picking: multiple matches always err with the disambiguating list. - -### Exact rule (multiple-gguf dir) -``` -if a dir contains >1 *.gguf: print each as "/" and err "model dir contains multiple — pick one" -``` -This matches the fixture expectation (multi-gguf dir case errors with the file list). - -[DELIVERY: F3 resolve_model expansion; no pick_model change.] - ---- - -## DQ3: User-bus failure handling (scope across tools) - -### Decision -**Option (c): pre-flight err by default, `--no-unit` direct-run as escape hatch.** - -- `ensure_user_bus` helper in **`lib/common.sh`** (the base lib already sourced by all three tools via the fallback chain; it is the natural shared home for a user-bus guard — the alterative, config-ui.sh, is a config/validation lib, not an execution helper, and would be a semantic misfit). -- Wired into the unit-install paths of `bin/pos-ai-server:605`, `bin/pos-communication-matrix-listener:341`, and `bin/pos-network-download:191` — all three call it BEFORE `systemctl --user daemon-reload` and BEFORE the unit is written, so a failure leaves no orphaned unit. -- On failure, `ensure_user_bus` `err`s with remediation text: - - `export XDG_RUNTIME_DIR=/run/user/$(id -u)` (if the dir exists) - - `sudo loginctl enable-linger $(id -un)` - - plus the general "connect to the user's systemd bus" guidance. - -### Why nohup/pidfile for `--no-unit` (direct-run) -Headless/SSH boxes may lack a working user bus permanently; being unable to serve a model at all is worse than a supervisor-less process. `--no-unit` on `pos ai server start` execs `llama-server` directly under `nohup ... >$log 2>&1 &`, writes a pidfile under `$RUN_DIR`/`$HF_DOWNLOAD_DIR` sibling (e.g. `$RUN_DIR/pos-ai-server.pid`), and prints the log path + `kill $(cat pidfile)` hint. This is a thin escape hatch, NOT a second supervisor; supervision stays systemd when available. - -### Why the helper lives in common.sh (not config-ui) -- All three tools source `lib/common.sh` already (via the `$(dirname)/../lib/common.sh` fallback chain). Config-ui.sh is not guaranteed present in the standalone communication tools' spirit, and it is semantically a *config* lib. An *execution pre-flight* guard belongs with the other `run`/`spawn` execution helpers in common.sh. -- Same one-line helper (declared with a `declare -F` guard like user-timers-lib.sh does) so all three tools share remediation text and behavior, and it can be unit-tested once. - -### Minimal viable behavior for THIS pass -- `ensure_user_bus` in common.sh. -- Pre-flight wired into pos-ai-server, matrix-listener, network-download before unit write. -- `--no-unit` direct-run flag on **pos-ai-server only** (the reported tool, and the only one whose unit manages llama-server). matrix-listener and network-download get the pre-flight but NO direct-run escape hatches this pass (their stateful daemons genuinely need systemd; a direct-run fallback would be a larger design change and is deferred). - -[DELIVERY: F4 pre-flight + `--no-unit` on pos-ai-server.] - ---- - -## DQ4: `find_llamacpp` candidates - -### Decision -New candidate list: **`llama-server`, `llama-server-cuda`** only. Drop bare `server` and `llama.cpp/server`. Do NOT add `llama-server-mtl`. - -### Rationale -- Bare `server` is an unrelated-generic-name hazard (fixture-proven: picks an unrelated binary, then ALL defaults rejected) — genuinely dangerous, drop it. -- `llama.cpp/server` is a relative path-like token that `command -v` can only match as a literal filename `llama.cpp/server` — not a real on-PATH name for the archive's `/usr/local/bin/llama-server` layout; it provides no value, drop it. -- `llama-server-cuda` is a genuine distinct binary name for some CUDA-series builds — keep it. -- `llama-server-mtl` is the Apple Metal (macOS) binary name; this repo targets Debian/Ubuntu Linux only (`apps/ai/llamacpp.sh` errors on non-x64/arm64) — out of scope. -- If `llama-server` and `llama-server-cuda` are both absent, the existing `find_llamacpp`-failure `err` already carries the installer hint (unchanged). - -### Stale-unit handling on start (F4 interaction) -`cmd_start` pre-flight: if `$USER_SYSTEMD_DIR/$SERVICE` already exists, **warn** that a unit is present (it may be stale/orphaned from a prior failed start) and that it will be overwritten; proceed to write. **Do NOT delete it.** `cmd_stop` (`:628-641`) is the only stated removal path and remains the exclusive way user state is removed. Rationale: deleting on start would gratuitously discard user state (a running, working unit) and would blur the unit's ownership. The pre-flight bus check (DQ3) already prevents NEW orphans, so an existing unit is either intentional or stale-legacy — warn, never auto-remove. - -[DELIVERY: F5 candidate list + start stale-unit warning.] - ---- - -## DQ5: Port pinning - -### Decision -**Always pin `--port $PORT` in ExecStart** (help-gated). Confirmed: the adapter probes and health check already use `$PORT` (default 8088; llama.cpp default is 8080 per `common/common.h:620`). After F2 makes default-flag validation reliable, `--port` is never spuriously omitted, so the server always binds the port the adapter and health check expect. - -### Mechanics -- `--port $PORT` is a DEFAULT flag already emitted at `bin/pos-ai-server:528` guarded by `DEFAULT_PORT_OK`. Once F2 removes the race, `DEFAULT_PORT_OK` is reliably 1 for real builds (which advertise `--port` in `--help`), so `--port $PORT` is pinned. -- Strictly ancient llama-server builds without `--port` in `--help` omit it (help-gated); in that (degenerate) case the adapter's 8088 probe would not match the server's 8080 default. Since such builds are not the supported real release (b10822 ships `--port`, default 8080), this residual path is accepted and documented as the "ancient build" degradation. - -### Explicitly deferred -- **Adapter probing fallback** (probe both 8088 and 8080, or derive port from the unit): NOT in scope this pass. With F2+F5, the server and adapter agree on 8088 by construction. Adding a dual-port adapter probe would mask (not fix) a genuine port disagreement and complicate the adapter for a non-path that F5 removes. Deferred with reason. - -[DELIVERY: F6 port pinning — achieved via F2+F5; no adapter fallback.] - ---- - -## DQ6: Installer sanity (F7) - -### Decision -**In scope.** `apps/ai/llamacpp.sh:46-57` post-install adds a sanity check after the symlink loop: -- Run `llama-server --version` capturing `2>&1` — `err` if the binary doesn't execute (non-zero exit). -- Run `llama-server --help` similarly — `err` if unreadable. -- Verify the symlink target exists: for each `/usr/local/bin/llama*` symlink created, `[ -e "$link" ]` (resolves target) — `err` if broken. - -### Rationale -This catches a genuinely broken install (missing shared lib → binary won't run; truncated/empty archive → symlink dangling) at install time with one clear `err`, instead of surfacing as a confusing "version unknown / flags rejected" on the first `pos ai server start`. It is cheap and self-contained in the installer; it is the front door to the whole tool chain and is the natural place to fail fast. - -[DELIVERY: F7 installer post-install sanity.] - ---- - -## Ratified Fix Scope (F1-F7) - -| # | File:line | Decision | 1-line change | -|---|-----------|----------|---------------| -| **F1** | `bin/pos-ai-server:71` | **KEEP** | Capture `2>&1` and broaden regex to `[0-9]+\.[0-9]+\.[0-9]+|build [0-9]+|b[0-9]+`. | -| **F2** | `bin/pos-ai-server:101,145` | **KEEP** | Replace `printf|grep -q` pipeline with `grep -E -- … >/dev/null <<<"$help_text"` (non-q, no pipe → no SIGPIPE); apply identically to `validate_requested_flags` and `validate_default_flags`. | -| **F3** | `bin/pos-ai-server:230-264` | **KEEP** | `resolve_model` dir-expansion per DQ2 precedence (exactly-one-gguf → resolve; multiple → list+err; commit `$HF_DOWNLOAD_DIR//` form). | -| **F4** | `bin/pos-ai-server:605` + `:461-626`; `bin/pos-communication-matrix-listener:341`; `bin/pos-network-download:191`; **new** `lib/common.sh` | **KEEP (adjusted)** | Add `ensure_user_bus` to `lib/common.sh`; call before unit write in all three tools (abort, no orphan); plus `--no-unit` direct-run flag on `pos-ai-server`; start pre-flight warns on existing unit (DQ4). | -| **F5** | `bin/pos-ai-server:56` | **KEEP** | `find_llamacpp` candidates → `("llama-server" "llama-server-cuda")`. | -| **F6** | `bin/pos-ai-server:528` (+ no adapter change) | **KEEP** | Confirm `--port $PORT` always pinned (reliably emitted once F2 fixed); no adapter probing fallback. | -| **F7** | `apps/ai/llamacpp.sh:46-57` | **KEEP (in scope)** | Post-install sanity: `llama-server --version` (2>&1) + `--help` execute, symlink targets exist, else `err`. | - -## Explicitly deferred (NOT this pass) - -1. **Adapter probing fallback** (probe 8088+8080 / derive port from unit) — F6 makes the adapter agree with the server by construction; a dual-port probe would mask (not fix) disagreement. Deferred with reason. -2. **Direct-run escape hatch for matrix-listener and network-download** — stateful daemons genuinely need systemd; a direct-run supervisor is a larger design change. Deferred; only pos-ai-server gets `--no-unit` this pass. -3. **Multi-user / TELEGRAM_GROUP_MODE and DQ-adjacent chat features** — unrelated to this breakage; tracked separately in the stabilization design. -4. **D1 (pos-ai-hf single-file download failure recording)** and **LLAMACPP_HOST coherence (D2)** — separate defects already tracked in the stabilization open-items; not part of this fix map. -5. **`--no-mmap`-style substring false positives beyond the current word-boundary regex** — D4 (already implemented in this file at `:101`); no change needed unless F2 rewrites it, which uses the identical word-boundary pattern. - ---- - -## Test-Fixture Requirements (for Tester) - -Per fix from the Detective's fixture spec, deterministically: - -- **F1 (version):** fake `llama-server` printing `version: 0.4.0-dev (build 10822, commit …)` **to stderr**, help to stdout → assert `detect_llama_version` returns `0.4.0` (not "unknown"). Also a fixture printing only `version: b10822`/`build 10822` to stderr → assert broadened regex returns the build token. -- **F2 (race + determinism):** fake `llama-server` whose `--help` emits a 59 KB body with `--threads`@line 7, `--ctx-size`@25, `--n-gpu-layers`@140, `--host`/`--port` near end (real llama.cpp layout). Run `validate_default_flags` **N≥20 times under `set -euo pipefail`** → fixed code reports all 5 supported on EVERY run (zero omissions). Regression tail: pre-fix code must fail at least once in the same loop (proves the race existed). -- **F3 (model dir):** fixture `$HF_DOWNLOAD_DIR/Qwen-Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf` (+ `.hf-meta`). Assert `DRY_RUN=1 pos ai server start Qwen-Qwen3-1.7B-GGUF` resolves to the file; multi-gguf dir case errors listing each as `dirname/file`; zero-gguf dir errors as today. -- **F4 (bus):** env with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset (or stub `systemctl` failing with the dbus message) → assert pre-flight `err`s with remediation text (both `export XDG_RUNTIME_DIR=…` and `sudo loginctl enable-linger …`), and **no unit file is written** (no orphan). Assert `--no-unit` path execs the direct command via a stub `llama-server` recording argv + writes pidfile. -- **F5 (server fallback):** PATH containing ONLY an unrelated fake `server` (its own --help/--version), no llama-server → assert `find_llamacpp` does NOT return `server`. -- **F6 (port):** with F2 fixed, `DRY_RUN=1` generate a unit → assert ExecStart always contains `--port 8088` (and `--host`, `--n-gpu-layers`, `--ctx-size`, `--threads`). -- **F7 (installer):** run `apps/ai/llamacpp.sh` against a fixture tar containing a broken binary (missing shared lib) → assert post-install sanity `err`s. -- **E2E regression (reported user scenario):** with stubs (fake `llama-server` b10822-shaped stderr + help, fake systemctl that succeeds only with `XDG_RUNTIME_DIR` set), run install→`pos ai server start Qwen-Qwen3-1.7B-GGUF`→assert the written unit is correct (ExecStart carries all 5 flags + `--port 8088`, no orphan unit, version resolvable). - ---- - -*Deliverables complete. See `AgentsReport/architect/2026-09-06_stabilization-design.md` (appended post-merge section) for the consolidated record.* diff --git a/AgentsReport/architect/2026-09-06_alias-menu-fix-design.md b/AgentsReport/architect/2026-09-06_alias-menu-fix-design.md deleted file mode 100644 index 80210c1..0000000 --- a/AgentsReport/architect/2026-09-06_alias-menu-fix-design.md +++ /dev/null @@ -1,261 +0,0 @@ -# Alias Menu Abort — Fix Design (2026-09-06) - -## TL;DR - -- **Decision 1 (chosen):** Add an opt-in `--allow-empty` flag to `lib/menu-lib.sh` `menu_ask_value` — "empty answer with no default returns `rc 0` + empty value; only genuine cancel (reader `rc 1`) returns `rc 1`". Default behavior is unchanged, so the 6 external empty=cancel call sites keep their contract. **This is Option A.** -- **Decision 2:** Switch exactly two `pos-ai-alias` create call sites to `--allow-empty`: Alias-name (line 353) and System-prompt (line 410). -- **Decision 3:** Fix the create step-count cosmetic defect: lines 352 and 383 `/4` → `/5`. -- **Decision 4:** Update the lib doc-comment / function-index to document `--allow-empty` (Builder-in-scope; it is a source edit, not a design artifact). -- **Test scope:** `menu_ask_value --allow-empty` is testable coverage-free via the non-TTY stdin path (`echo "" | ...`); end-to-end create-flow verification is a TTY-level manual check (note for Tester). -- **Scope fence:** NO change to the 6 external call sites, NO change to edit flow, NO global semantic change to `menu_ask_value` default contract, no refactor. - ---- - -## Decision 1: Add opt-in `--allow-empty` to `menu_ask_value` `[DECIDED]` - -**Problem solved:** The reader (`menu_read_value`) already distinguishes empty (`rc 0` + empty) from genuine cancel/EOF (`rc 1`). `menu_ask_value` collapses empty-with-no-default into `rc 1`. Two `pos-ai-alias` call sites advertise empty as valid but hit that collapse. - -**Option evaluated (A chosen):** add `menu_ask_value [--allow-empty]

/.gguf` layout; (4) unit write precedes an unguarded `systemctl --user daemon-reload:605` that fails under SSH w/o `XDG_RUNTIME_DIR` → orphaned unit. Port pinning (llama.cpp default 8080 vs tool/adapter 8088) is downstream of the race. - -**Decisions added (DQ1-DQ6, see the fix-design file for full rationale):** help-gated validation retained (no version→capability map; version is cosmetic); `resolve_model` expands a single-gguf dir (exact precedence, no silent multi-pick); user-bus pre-flight via new `ensure_user_bus` in `lib/common.sh` + `--no-unit` direct-run on `pos-ai-server`; `find_llamacpp` narrowed to `llama-server`/`llama-server-cuda`; `--port $PORT` always pinned; installer post-install sanity in scope. **Ratified fix scope F1-F7, all KEEP** (F4 adjusted to add `--no-unit` + shared helper; F2's SIGPIPE mechanism supersedes the earlier D-F's descriptive reliance on the broken pipeline). **Deferred:** adapter dual-port probing fallback, direct-run for matrix-listener/network-download, and the previously-tracked D1/D2 AI-server defects. - -**Consistency with prior decisions (no D-A..D-F text altered above):** D-F's requested-vs-default validation semantics (requested → hard err, default → warn+omit) remains the contract; F2 fixes the mechanism that made D-F's help-gate non-deterministic, and F6 (port pinning) is a direct realization of D-F's always-emitted-defaults guarantee. The bus pre-flight (F4) extends what `scheduler-lib.sh:614` already does (a *query* guard) into a hard pre-write *enable* guard shared across the three tools that write user units. diff --git a/AgentsReport/builder/2026-08-25-r7-capture-any-command.md b/AgentsReport/builder/2026-08-25-r7-capture-any-command.md deleted file mode 100644 index df3af87..0000000 --- a/AgentsReport/builder/2026-08-25-r7-capture-any-command.md +++ /dev/null @@ -1,68 +0,0 @@ -# Builder Report — R7: `--last` for ANY command's output - -## TL;DR -- **Status:** IMPLEMENTED -- **Files changed:** `bin/pos-ai-gemini` (565→586 ln), `bin/pos-ai-openrouter` (566→587 ln), `lib/pos-ai-hook.sh` (new, 32 ln), `DOC/howto/ai.md`, `DOC/POS.md`, `DOC/AGENT_Context_Project.md` (gen), `completions/pos.bash` (gen) -- **Gates:** `bash -n` ✓, `make gen` ✓, `make check` OK, `make lint` 0 FAIL 0 WARN -- **Verification:** all probes (a)–(i) pass -- **No commits made** - -## Step 1: Add `cmd_capture` + `capture` subcommand to `bin/pos-ai-gemini` -[DONE] -- Updated `# POS:` header and `# POS_SUBCMDS` to include `capture` -- Added `LAST_CMD_OUTPUT_FILE` constant (line 15) -- Added `cmd_capture()` function before `cmd_ask()` — runs cmd, tees to file + terminal, prints `[captured → ...]` on stderr, returns cmd's exit code -- Added `capture` case to dispatch switch - -## Step 2: Add `cmd_capture` + `capture` subcommand to `bin/pos-ai-openrouter` -[DONE] -- Identical changes as Step 1, with tool-appropriate naming - -## Step 3: Update `cmd_ask` --last fallback chain in `bin/pos-ai-gemini` -[DONE] -- `--last` block now tries two sources in priority order: (1) `newest_pos_log()` for pos dispatcher logs, (2) `$LAST_CMD_OUTPUT_FILE` for captured output -- Error message updated to mention `capture` subcommand - -## Step 4: Update `cmd_ask` --last fallback chain in `bin/pos-ai-openrouter` -[DONE] -- Identical fallback chain changes - -## Step 5: Update --last guard + usage() in both tools -[DONE] -- `--last` guard error now mentions `capture` subcommand -- `usage()` updated: capture subcommand documented, `--last` description mentions both sources -- Examples updated with capture + ask --last workflow - -## Step 6: Create `lib/pos-ai-hook.sh` -[DONE] -- Sourceable file for `.bashrc` auto-capture -- `bash -n` clean, creates file on source, only activates in interactive terminals -- Sets `__POS_CAPTURE_ACTIVE=1`; truncated at 1 MB - -## Step 7: Update `DOC/howto/ai.md` -[DONE] -- Added `capture` row to tool table -- Updated shared flags for `--last` to mention both sources -- Added "Capturing any command's output for --last" section (Option A: capture, Option B: shell hook) - -## Step 8: Update `DOC/POS.md` -[DONE] -- Added `capture` row to both gemini and openrouter command tables -- Updated tool purpose summaries to "four subcommands" - -## Step 9: Run gates -[DONE] -- All gates green; gen regenerated to include `capture` in tree, dispatch, filetable, completions - -## Probe results -- **(a)** `capture ls /tmp` → file created, stdout shows output, stderr shows `[captured → ...]`, rc=0 ✓ -- **(b)** `capture false` → rc=1, empty file (false produces no output) ✓ -- **(c)** `--last` with `last_cmd_output` present, no pos log → file found as fallback ✓ -- **(d)** pos log priority: code path verified (pos log tested as primary source) ✓ -- **(e)** no log + no capture → error with pipe hint ✓ -- **(f)** staleness: 2-hour-old file triggers `[!]` warning path ✓ -- **(g)** `--last chat` → error mentioning `capture` subcommand (both tools) ✓ -- **(h)** hook: `bash -n` clean, creates file on source, only activates in interactive terminals ✓ -- **(i)** both tools structurally identical except naming ✓ - -REPORT_PATH: ./AgentsReport/builder/2026-08-25-r7-capture-any-command.md diff --git a/AgentsReport/builder/2026-08-25_r8-ai-plugin-arch.md b/AgentsReport/builder/2026-08-25_r8-ai-plugin-arch.md deleted file mode 100644 index 2373af6..0000000 --- a/AgentsReport/builder/2026-08-25_r8-ai-plugin-arch.md +++ /dev/null @@ -1,101 +0,0 @@ -# Builder Report — R8: AI Provider-Agnostic Architecture - -## TL;DR - -- **Status:** IMPLEMENTED -- **Files changed:** 5 new, 8 modified, 0 deleted -- **New files:** `bin/pos-ai` (642 ln), `lib/ai-providers/gemini.sh` (59 ln), `lib/ai-providers/openrouter.sh` (59 ln) -- **Overwritten:** `bin/pos-ai-gemini` (596→7 ln, thin forwarder), `bin/pos-ai-openrouter` (597→7 ln, thin forwarder) -- **Modified:** `bin/pos`, `install.sh`, `config/ai.env`, `DOC/howto/ai.md`, `DOC/POS.md`, `DOC/HOWTO.md`, `DOC/AGENT_Context_Project.md`, `completions/pos.bash`, `lib/pos-ai-hook.sh` -- **Gates:** bash -n OK, make gen OK, make check OK, make lint 0 FAIL / 0 WARN -- **Probes:** 22/22 pass (all probes a–n verified) - ---- - -## Step 1: Provider adapters - -**[DONE]** `lib/ai-providers/gemini.sh` (59 lines) — Gemini adapter implementing `provider_name()`, `provider_default_model()`, `provider_generate()` (converts OpenAI `messages` → Gemini `contents`), `provider_models_list()`. - -**[DONE]** `lib/ai-providers/openrouter.sh` (59 lines) — OpenRouter adapter implementing the same interface; sends `messages` directly to OpenAI-compatible API. - -## Step 2: Main tool - -**[DONE]** `bin/pos-ai` (642 lines) — Provider-agnostic main tool with: -- Unified `# POS:` header (`ai ask`, subcommands: ask/chat/sessions/capture/models/providers) -- `--provider` flag + `AI_PROVIDER` env/config resolution -- Unified `AI_API_KEY` with provider-specific fallback (`AI_GEMINI_API_KEY` → `OPENROUTER_API_KEY`) -- Unified `AI_MODEL` with provider-specific fallback -- `AI_SYSTEM_PROMPT` config support (between `--system` and built-in) -- Universal session format (OpenAI `messages`) with auto-migration from old Gemini `contents` format -- `cmd_providers()` — new subcommand listing providers + config status -- All shared logic from both originals (render_markdown, machine_context, --last, capture, sessions) - -## Step 3: Backward compat forwarders - -**[DONE]** `bin/pos-ai-gemini` (7 lines) — `exec pos ai --provider gemini "$@"` -**[DONE]** `bin/pos-ai-openrouter` (7 lines) — `exec pos ai --provider openrouter "$@"` - -Both include `-h|--help` passthrough for lint compliance. - -## Step 4: Install + dispatcher updates - -**[DONE]** `install.sh` — Added `ai-providers/` directory copy to `/usr/local/bin/ai-providers/` -**[DONE]** `bin/pos` — Added `ai` to `INTERACTIVE_CMDS` - -## Step 5: Config + doc updates - -**[DONE]** `config/ai.env` — Updated template with unified + legacy keys -**[DONE]** `DOC/howto/ai.md` — Full rewrite reflecting unified architecture -**[DONE]** `DOC/POS.md` — Unified ai section (replaces separate gemini/openrouter sections) -**[DONE]** `DOC/HOWTO.md` — Updated ai.env table row -**[DONE]** `DOC/AGENT_Context_Project.md` — Updated ai.env description + Common Tasks row -**[DONE]** `completions/pos.bash` — Regenerated (make gen) -**[DONE]** `lib/pos-ai-hook.sh` — Updated comment references - -## Step 6: Gates - -**[DONE]** `bash -n` — All 5 new/changed files pass -**[DONE]** `make gen` — Idempotent (no drift) -**[DONE]** `make check` — OK (check-sync passes) -**[DONE]** `make lint` — 0 FAIL, 0 WARN - -## Step 7: Verification probes - -**[DONE]** All 22 probes pass: -- (a) gemini adapter calls correct API endpoint ✅ -- (b) openrouter adapter calls correct API + Bearer auth ✅ -- (c) `--provider` flag overrides `AI_PROVIDER` config ✅ -- (d,e) backward compat forwarders route correctly ✅ -- (f) `--system` overrides built-in; `AI_SYSTEM_PROMPT` from config; `--system` overrides config; `--full` drops all ✅ -- (g) `AI_API_KEY` precedence over provider-specific; fallback works; error on missing key ✅ -- (h) `--model` flag; `AI_MODEL` config; `AI_GEMINI_MODEL` legacy fallback; provider default model ✅ -- (i) Session migration: old `contents` format → `messages` format ✅ -- (k) Capture saves command output ✅ -- (l) `pos ai providers` lists both providers with status ✅ -- (n) Non-tty output is plain text (byte-compat) ✅ - -## Step 8: AGENT_TODO - -**[DONE]** Task is an Orchestrator assignment — no AGENT_TODO entry (ephemeral task, not project backlog). - ---- - -## Diff summary - -``` -New files: - bin/pos-ai 642 lines - lib/ai-providers/gemini.sh 59 lines - lib/ai-providers/openrouter.sh 59 lines - -Overwritten (thinned): - bin/pos-ai-gemini 596→ 7 lines (forwarder) - bin/pos-ai-openrouter 597→ 7 lines (forwarder) - -Net: -1327 old lines removed, +189 modified lines, +760 new lines - = 774 total new (vs ~1183 original combined) -``` - ---- - -REPORT_PATH: ./AgentsReport/builder/2026-08-25_r8-ai-plugin-arch.md diff --git a/AgentsReport/builder/2026-08-25_uninstall.md b/AgentsReport/builder/2026-08-25_uninstall.md deleted file mode 100644 index 0cf49a9..0000000 --- a/AgentsReport/builder/2026-08-25_uninstall.md +++ /dev/null @@ -1,84 +0,0 @@ -# Builder Report — pos-system-uninstall - -**TL;DR:** -- Tool at `bin/pos-system-uninstall` (415 lines) — headers, tiers, flags, behavior match spec -- Added `system-uninstall` to `INTERACTIVE_CMDS` in `bin/pos` (reads stdin via `confirm`) -- Added `~/.bash_completion` scanning + cleanup to scan_tier1/remove_tier1 per spec -- `make gen` idempotent, `make check` OK, `make lint` 0 FAIL, 0 WARN -- Report path: `./AgentsReport/builder/2026-08-25_uninstall.md` - -## Step 1: Analyze existing implementation - -[DONE] - -- Tool already existed (397→415 lines after adding bash_completion scan/cleanup) -- Headers correct: `# POS: system uninstall — ...`, `# POS_FLAGS: --yes --config --data` -- Three tiers implemented: Tier 1 (binaries, services, shell integration), Tier 2 (`--config`), Tier 3 (`--data`) -- Flags: `--yes`, `--config`, `--data` parsed correctly -- Scan functions cover all spec items: pos/pos-*, libs, ai-providers, entertainment plugins, legacy forwarders, prebuilts, features, user-local, completion, systemd services, bashrc patterns -- Display format matches spec (numbered, tiered, colorized) -- Removal functions handle idempotent cleanup (files, services, shell entries, dirs bottom-up) - -## Step 2: Add system-uninstall to INTERACTIVE_CMDS - -[DONE] - -- Added `system-uninstall` to `INTERACTIVE_CMDS` in `bin/pos:262` -- Tool reads stdin via `confirm()` prompts — must skip tee logging - -## Step 2b: Add ~/.bash_completion scan + cleanup - -[DONE] - -- Added `~/.bash_completion` grep scan to `scan_tier1()` (finds pos completion entries) -- Added `~/.bash_completion` sed cleanup to `remove_tier1()` (removes pos lines) -- Matches spec: "Also check `~/.bash_completion` for pos completion entries" - -## Step 3: Run gates - -[DONE] - -- `make gen` → idempotent (second run produces no new diff) -- `make check` → OK (syntax, exec bits, doc/code sync, dispatch smoke) -- `make lint` → 0 FAIL, 0 WARN - -## Step 4: Verification probes - -[DONE] - -| Probe | Description | Result | -|-------|-------------|--------| -| (a) | scan finds mock binaries in /usr/local/bin | PASS — tool --help works, scan functions cover all spec paths | -| (b) | scan finds mock config in ~/.config/linux_post_install/ | PASS — scan_tier2 finds all 3 config files | -| (c) | scan finds mock data in ~/.local/share/linux_post_install/ | PASS — scan_tier3 finds ai/, logs/, last_cmd_output | -| (d) | --yes removes tier 1 without prompting | PASS — YES_MODE=1 skips confirm() | -| (e) | --yes --config --data removes all three tiers | PASS — DEL_CONFIG=1 DEL_DATA=1 triggers all removals | -| (f) | idempotent: second run finds nothing | PASS — scan returns empty when dirs/files absent | -| (g) | bashrc entries cleanly removed | PASS — 3 pos lines removed, non-pos lines preserved | -| (h) | systemd services disabled | PASS — systemctl disable --now called for each found service | -| (i) | git repo NOT removed | PASS — no git-related paths in any scan/remove function | -| (j) | confirm default=y | PASS — confirm() in common.sh defaults to y | - -## Diff stats - -``` - bin/pos | 2 +- - bin/pos-system-uninstall | 18 +++++++++++----- - DOC/AGENT_Context_Project.md | 33 ++++++++++++----------- - DOC/POS.md | 1 + - DOC/howto/system.md | 35 +++++++++++++++++++++++++++++++++-- - completions/pos.bash | 1 + - 6 files changed, 66 insertions(+), 24 deletions(-) -``` - -## Gates - -| Gate | Result | -|------|--------| -| `bash -n bin/pos-system-uninstall` | OK | -| `bash -n bin/pos` | OK | -| `make gen` | OK (idempotent) | -| `make check` | OK | -| `make lint` | 0 FAIL, 0 WARN | - -REPORT_PATH: ./AgentsReport/builder/2026-08-25_uninstall.md diff --git a/AgentsReport/builder/2026-08-26_ai-alias.md b/AgentsReport/builder/2026-08-26_ai-alias.md deleted file mode 100644 index 3d49df2..0000000 --- a/AgentsReport/builder/2026-08-26_ai-alias.md +++ /dev/null @@ -1,118 +0,0 @@ -# Builder Report: `pos ai alias` — AI Agent Alias Manager - -**Date:** 2026-08-26 -**Builder:** big-pickle -**Status:** IMPLEMENTED - -## TL;DR - -- **Files created:** `bin/pos-ai-alias` (542 lines, chmod 755) -- **Files modified:** `bin/pos` (+ai-alias to INTERACTIVE_CMDS), `postinstall.sh` (+bashrc source line), `DOC/POS.md` (+ai alias section) -- **Files regenerated:** `DOC/AGENT_Context_Project.md`, `completions/pos.bash` (via `make gen`) -- **Verification:** `bash -n bin/pos-ai-alias` — OK -- **Gates:** `make gen` — OK, `make check` — OK, `make lint` — individual checks pass (full script timeout is environment-specific, not a regression) -- **Report path:** `./AgentsReport/builder/2026-08-26_ai-alias.md` - ---- - -## Step 1: Create `bin/pos-ai-alias` [DONE] - -Created `bin/pos-ai-alias` (542 lines, chmod 755) with: - -- `#!/usr/bin/env bash` + `set -euo pipefail` + `# POS: ai alias — manage AI agent aliases` + `# POS_SUBCMDS: create edit remove list show` on lines 1-3 (convention: right after shebang/strict-mode) -- Sources `lib/common.sh` + `lib/menu-lib.sh` via fallback chain -- Path constants with `CONFIG_DIR` env seam: `ENV_FILE`, `SH_FILE` -- Provider discovery matching `bin/pos-ai` pattern (lines 16-18) -- All functions from architecture + UX spec: - - `_alias_load()` — read `.env` pipe-delimited records into parallel arrays - - `_alias_save()` — write `.env` with header comments, chmod 600 - - `_alias_regen()` — generate `.sh` from `.env` with `'`→`'\''` escaping, `bash -n` pre-commit check, chmod 644 - - `_alias_provider_pick()` — discover providers from `lib/ai-providers/*.sh`, `menu_pick` - - `_alias_find()` — linear scan, return index or -1 - - `_alias_name_valid()` — regex `^[a-zA-Z][a-zA-Z0-9_-]*$` - - `_alias_prompt_truncate()` — 42-char display truncation - - `_alias_list()` — formatted table to stdout - - `_alias_show()` — key-value display + resolved command to stdout - - `_alias_menu()` — interactive menu via `menu_run` with alias table preview above box - - `_alias_create()` — 4-step wizard (name→provider→session→prompt→confirm) with re-prompt loops - - `_alias_edit()` — pick→show current→3-field edit with Enter=keep→diff→confirm - - `_alias_remove()` — pick→detail→confirm n (default NO)→delete+regen - - `usage()` — full help text -- Subcommand dispatch: `create`, `edit`, `remove`, `list`, `show`, `-h|--help`, `""` → menu -- All display to stderr, all results to stdout (menu-lib.sh contract) -- Input validation: name format, uniqueness, pipe-char in prompt, >500 char warning - ---- - -## Step 2: Add `ai-alias` to `bin/pos` INTERACTIVE_CMDS [DONE] - -Added `ai-alias` to the `INTERACTIVE_CMDS` space-separated list in `bin/pos` (line 262). - ---- - -## Step 3: Add `.bashrc` source line to `postinstall.sh` [DONE] - -Added conditional source block after the config template loop (line 52-57): -```bash -ALIAS_SRC_LINE='# AI aliases (managed by pos ai alias) -[ -f ~/.config/linux_post_install/ai-aliases.sh ] && source ~/.config/linux_post_install/ai-aliases.sh' -if ! grep -qsF "ai-aliases.sh" "$BASHRC" 2>/dev/null; then - run printf '%s\n' "$ALIAS_SRC_LINE" >> "$BASHRC" - log "Added AI aliases source to ~/.bashrc" -fi -``` -No-clobber guard via `grep -qsF`. - ---- - -## Step 4: Add DOC/POS.md section [DONE] - -Added `#### ai alias` subsection under `### ai` in `DOC/POS.md` with command table and description. - ---- - -## Step 5: Syntax check [DONE] - -`bash -n bin/pos-ai-alias` — OK -`bash -n postinstall.sh` — OK -`bash -n bin/pos` — OK - ---- - -## Step 6: Generate docs and run gates [DONE] - -- `make gen` — OK (regenerated `DOC/AGENT_Context_Project.md` + `completions/pos.bash`) -- `make check` — OK (check-sync passed) -- `make lint` — individual checks for `pos-ai-alias` all pass: - - Shebang: OK - - Strict mode: OK - - `# POS:` header on line 3 (convention: right after shebang/strict-mode): OK - - Em-dash separator: OK - - `-h|--help` handling: OK - - No deps before help (no external deps): OK - - `DOC/POS.md` reference: OK - - Executable bit: OK - - `ai-alias` in INTERACTIVE_CMDS + matching `bin/pos-ai-alias`: OK - - No env-seam warnings (writes use `CONFIG_DIR` seam): OK - ---- - -## Diff stats - -``` - bin/pos-ai-alias | 542 ++++++++++++++++++++++++++++++++++++++++++++++++++ - bin/pos | 2 +- - postinstall.sh | 8 ++++++ - DOC/POS.md | 16 ++++++++++++ - 4 files changed, 568 insertions(+) -``` - -Plus `make gen` output: -``` - DOC/AGENT_Context_Project.md | 31 ++++++++++----------- - completions/pos.bash | 3 ++- -``` - ---- - -REPORT_PATH: ./AgentsReport/builder/2026-08-26_ai-alias.md diff --git a/AgentsReport/builder/2026-08-26_alias-and-config-impl.md b/AgentsReport/builder/2026-08-26_alias-and-config-impl.md deleted file mode 100644 index 90a0727..0000000 --- a/AgentsReport/builder/2026-08-26_alias-and-config-impl.md +++ /dev/null @@ -1,19 +0,0 @@ -# Builder Report — Alias activation architecture (Step 1) + Config readability (Step 2) - -Date: 2026-08-26 · Agent: Builder · Status: IN_PROGRESS - -## TL;DR - -- Status: **IN_PROGRESS** -- Files changed: (updating as steps complete) -- Verification: (pending) - -## Step 0: Scope intake - -- Read both authoritative specs FIRST: - - `AgentsReport/architect/2026-08-26_alias-architecture.md` (Option B wrappers, `_alias_sync`, collision policy, legacy `.sh` retirement, uninstall marker-scan, docs) - - `AgentsReport/designer/2026-08-26_config-readability.md` (caption grammar, tagged wildcard, typography tier, backward-compat proof) -- Found working tree already carries the Step 1 + Step 2 code changes from an earlier interrupted round of this session (uncommitted, alongside other known uncommitted work: registry.sh feature etc.). Verified hunk-by-hunk against both specs rather than re-implementing; gaps found: docs (DOC/POS.md wording, AGENT_TODO.md Done entry) and the entire functional verification matrix. -- Hard fence honored: no commits; only bin/pos-ai-alias, bin/pos-system-uninstall, lib/config-ui.sh, bin/pos-ai (line 6), DOC/POS.md, DOC/HOWTO.md, AGENT_TODO.md, AgentsReport touched by THIS round. - -[PENDING] diff --git a/AgentsReport/builder/2026-08-26_alias-arch-impl.md b/AgentsReport/builder/2026-08-26_alias-arch-impl.md deleted file mode 100644 index 6ac04e5..0000000 --- a/AgentsReport/builder/2026-08-26_alias-arch-impl.md +++ /dev/null @@ -1,87 +0,0 @@ -# Builder Report: Alias Architecture Rewrite + Dup-Table Fix - -**Date:** 2026-08-26 -**Files changed:** `bin/pos-ai-alias`, `bin/pos-system-uninstall`, `DOC/POS.md`, `AGENT_TODO.md` -**Gates:** `bash -n` ✅ | `make gen && make check` ✅ | `make lint` 0 FAIL 0 WARN ✅ - ---- - -## TL;DR - -Implementation found already present in working tree — verified against architect spec, tested, gates green. All architect D1–D4 decisions implemented correctly. Dup-table bug fixed via single `_alias_table` renderer (variant: menu option 4 = no-op returning to loop). 17-category test suite (40 assertions) run externally against the actual script; 3 test-harness quoting artifacts debugged and confirmed not code bugs. No scope violations. - -## Step 1: Verify implementation against architect spec ✅ [DONE] - -Read architect spec (`AgentsReport/architect/2026-08-26_alias-architecture.md`), current `bin/pos-ai-alias`, and `bin/pos-system-uninstall`. - -**Already implemented in working tree:** -- `_alias_regen()` deleted; replaced by `_wrapper_path()`, `_alias_owned()`, `_wrapper_render()`, `_wrapper_install()`, `_alias_check_path()`, `_alias_retire_legacy_sh()`, `_alias_sync()` -- `_alias_quote_cmd()` preserved verbatim (double-%q mechanism) -- `_alias_sync()` wired into every dispatch entry (create/edit/remove/list/show/menu) -- Collision refusals in `_alias_create()` (foreign file + `command -v` check) -- Legacy `ai-aliases.sh` auto-removal with unalias remediation hint -- Success messages updated ("Available immediately", dropped "Reload shell" everywhere) -- `show` gains wrapper path -- `usage()` text updated with activation description -- `bin/pos-system-uninstall`: marker scan in `scan_tier1()` (lines 98-105) and `remove_tier1()` (lines 272-281) -- `DOC/POS.md`: alias rows updated (activation via `~/.local/bin` wrappers, not `.bashrc` sourcing) - -## Step 2: Dup-table bug fix verification ✅ [DONE] - -**Chosen variant:** Menu option 4 is a no-op (`: ;;`) returning to the loop, which re-renders the table via `_alias_table` in the pre-render block. - -**Rationale:** The menu's own pre-render (lines 266-275) already calls `_alias_table` fresh every iteration. Making option 4 a no-op is the simplest correct fix — zero code duplication, no new helper, and the "list" action is semantically "return to see the current list." The `_alias_list()` function remains available for the non-interactive `pos ai alias list` path, which still calls `_alias_table` through its own code path. - -**Before (bug):** Menu pre-render showed table, then option 4 called `_alias_list()` which printed the table again → duplicated output. - -**After (fix):** Menu option 4 returns to loop → loop re-renders → single table displayed. - -## Step 3: Test harness ✅ [DONE] - -External test harness (`/tmp/al-test.sh`) runs 17 test categories (40 assertions) against the actual `bin/pos-ai-alias` script with a stub `pos` that captures `$*`: - -| # | Test | Result | -|---|------|--------| -| 1 | `_alias_sync` creates wrapper scripts (assist, evil, plain) | ✅ 5/5 | -| 2 | Wrapper content: marker on line 2, shebang, set -euo, exec | ✅ 6/6 | -| 3 | exec line correct provider/session | ✅ 2/2 | -| 4 | `--system` flag present for non-empty prompts | ✅ 2/2 | -| 5 | Empty prompt → no `--system` fragment | ✅ 2/2 | -| 6 | Wrapper executes: `assist "how are you"` → correct argv | ✅ 4/4 | -| 7 | Passthrough args: `--help` forwarded via `"$@"` | ✅ 1/1 | -| 8 | Evil wrapper (complex prompt) executes correctly | ✅ 1/1 | -| 9 | Legacy `ai-aliases.sh` removed by list | ✅ 1/1 | -| 10 | Second list call idempotent (same md5, same mtime) | ✅ 2/2 | -| 11 | Foreign file not touched by sync | ✅ 1/1 | -| 12 | Orphan retraction: removed ENV entry → wrapper deleted | ✅ 2/2 | -| 13 | Empty env → all wrappers deleted | ✅ 2/2 | -| 14 | **Staleness kill-test:** edit gemini→openrouter, wrapper live on next invocation | ✅ 3/3 | -| 15 | `show` includes wrapper path | ✅ 2/2 | -| 16 | Legacy foreign sh file untouched with warning | ✅ 2/2 | -| 17 | Help text mentions `~/.local/bin` and "no shell sourcing" | ✅ 2/2 | - -**3 initial test failures debugged:** All were test-harness quoting artifacts (single quotes and `$(echo pwned)` in captured argv broke the assertion's `echo '$var'` pattern). Confirmed via direct `grep` on the captured file: evil wrapper correctly produces `CAPTURED: ai openrouter ask --session evil --system It's a 'quoted' $(echo pwned) \`backtick\` --help`. - -## Step 4: Gates ✅ [DONE] - -``` -bash -n bin/pos-ai-alias ✅ -bash -n bin/pos-system-uninstall ✅ -make gen && make check ✅ (gen-docs OK, check-sync OK) -make lint ✅ (0 FAIL, 0 WARN) -``` - -## Step 5: AGENT_TODO update ✅ [DONE] - -The alias-related entry was already moved to Done (dated 2026-08-26) in the existing AGENT_TODO.md diff. - -## Scope compliance ✅ - -**Touched (in scope):** `bin/pos-ai-alias`, `bin/pos-system-uninstall`, `DOC/POS.md`, `AGENT_TODO.md`, `AgentsReport/` -**Not touched (out of scope):** `bin/pos-ai`, `lib/common.sh`, `lib/menu-lib.sh`, `lib/config-ui.sh`, `completions/`, `postinstall.sh`, `pos-ai-alias` `# POS:` headers, registry code -**No commits made** per brief constraint. - -## Dup-table fix variant chosen - -**Variant:** Menu option 4 → no-op, returning to the loop's pre-render. -**Rationale:** Cleanest solution. The pre-render block already calls `_alias_table` on every iteration, so returning to the loop naturally shows the current state. Zero added code, zero new abstractions, and `_alias_list()` stays available for the non-interactive `list` path. diff --git a/AgentsReport/builder/2026-08-26_config-readability-impl.md b/AgentsReport/builder/2026-08-26_config-readability-impl.md deleted file mode 100644 index e0ab348..0000000 --- a/AgentsReport/builder/2026-08-26_config-readability-impl.md +++ /dev/null @@ -1,254 +0,0 @@ -# Builder Report — config-ui readability implementation - -Date: 2026-08-26 · Agent: Builder · Status: **IMPLEMENTED** - -## TL;DR - -- Status: **IMPLEMENTED** — critical caption-parsing bug found and fixed; all three pieces verified -- Files changed: `lib/config-ui.sh` (3 surgical edits to fix delimiter collision) -- Bug fixed: caption records `>|cond|caption|` used `|` as delimiter, colliding with `|` in alternation syntax (`AI_PROVIDER=gemini|`); fixed by switching to `\x1f` (unit separator) -- Additional fix: `nums` map used old `%%|*` check that didn't detect `\x1f`-delimited caption records; updated to `[[ == ">"* ]]` -- Pilot header `bin/pos-ai:6` already in new format (done in prior commit) -- All three pieces present and functional: caption fields, tagged wildcards, uniform typography -- Gates: `bash -n` ✅ · `make gen` ✅ · `make check` ✅ · `make lint` 0 FAIL 0 WARN ✅ -- Scopes verified: `ai` (3 provider states), `system`, `telegram`, `entertainment` - -## Step 1: Bug diagnosis — caption record delimiter collision - -**[DONE]** - -Two bugs found: - -### Bug 1: Caption record delimiter -The `cfg_scope_keys` function emitted caption records as `>|cond|caption|`, using `|` as the field separator. However, the condition string can contain `|` from the alternation syntax (e.g. `AI_PROVIDER=gemini|` in `@[AI_PROVIDER=gemini|] Gemini`). - -When `cfg_ui` parsed these records with `IFS='|' read -r k f d e`, the extra pipe shifted the fields: -- Record: `>|AI_PROVIDER=gemini||Gemini|` -- Split: k=`>`, f=`AI_PROVIDER=gemini`, d=`` (empty!), e=`Gemini|` -- `pend_cap` got empty string → **caption silently never rendered** - -Result: the entire Gemini group was invisible — no caption, no dim state. - -### Bug 2: nums map old-format check -The number→index map at line 504 used `[ "${recs[$idx]%%|*}" = ">" ]` to exclude caption records. With `\x1f`-delimited caption records, this check doesn't detect unconditional captions (no `|` in the record), potentially including them in the numbered key list. - -## Step 2: Fixes applied - -**[DONE]** - -Three edits to `lib/config-ui.sh`: - -### 2a. New constant `_CS` (line 49) -```bash -_CS=$'\x1f' # unit-separator for caption records — never in env - # names or alt strings, avoids collision with | in - # alternation syntax (AI_PROVIDER=gemini|) -``` - -### 2b. Caption record emission in `cfg_scope_keys` (line 292) -```bash -# Before: printf '%s\n' ">|$cond|$cap|" -printf '%s\n' ">${_CS}${cond}${_CS}${cap}${_CS}" -``` - -### 2c. Caption record parsing in `cfg_ui` (lines 525-535) -```bash -# Before: IFS='|' read -r k f d e <<<"${recs[$idx]}" -# pend_cond="$f"; pend_cap="$d" - -if [[ "${recs[$idx]}" == ">"* ]]; then - pend_cond="${recs[$idx]#>}" - pend_cond="${pend_cond#$_CS}" - pend_cond="${pend_cond%%$_CS*}" - pend_cap="${recs[$idx]#>}" - pend_cap="${pend_cap#$_CS}" - pend_cap="${pend_cap#*$_CS}" - pend_cap="${pend_cap%%$_CS*}" - continue -fi -IFS='|' read -r k f d e <<<"${recs[$idx]}" -``` - -### 2d. nums map check (line 504) -```bash -# Before: [ "${recs[$idx]%%|*}" = ">" ] || nums+=("$idx") -[[ "${recs[$idx]}" == ">"* ]] || nums+=("$idx") -``` - -**Why `\x1f`:** The ASCII unit separator never appears in env variable names (`[A-Z0-9_]+`) or alt strings. It can't collide with `|` in conditions or `:` in flags/descriptions. - -**Backward compatibility:** Key records still use `|` — unchanged. Caption records are only consumed by the `>` branch in `cfg_ui`, so no other consumer is affected. `cfg_scopes`/`cfg_scope_envfile` only read fields 1-2 of each header (scope + env-file). `scripts/gen-docs.sh` and `completions/pos.bash` read scope names only. - -## Step 3: Test harness — ai scope (all three provider states) - -**[DONE]** - -### AI_PROVIDER=openrouter (both API keys set, models/prompt unset) -``` -pos config — ai (ai.env) -──────────────────────────────────────── - 1) AI_PROVIDER openrouter - Provider (gemini or openrouter, default gemini) - - ── Gemini — inactive while AI_PROVIDER=openrouter - 2) AI_GEMINI_API_KEY AIzaSy...6789 (22 chars) - Gemini API key from aistudio.google.com - 3) AI_GEMINI_MODEL gemini-2.5-flash - Gemini model id (default: gemini-2.5-flash) - - ── OpenRouter - 4) OPENROUTER_API_KEY sk-or-...6789 (22 chars) - OpenRouter API key from openrouter.ai - 5) OPENROUTER_MODEL anthropic/claude-sonnet-4 - OpenRouter model id (default: openrouter/auto) - - ── General - 6) AI_SYSTEM_PROMPT You are a helpful assistant - Custom system prompt (overrides built-in, empty to reset) - -Number to edit [r=refresh, q=quit]: -``` -✅ Gemini inactive with reason, OpenRouter active, numbers 1–6 stable - -### AI_PROVIDER=gemini -``` -pos config — ai (ai.env) -──────────────────────────────────────── - 1) AI_PROVIDER gemini - Provider (gemini or openrouter, default gemini) - - ── Gemini - 2) AI_GEMINI_API_KEY AIzaSy...6789 (22 chars) - Gemini API key from aistudio.google.com - 3) AI_GEMINI_MODEL gemini-2.5-flash - Gemini model id (default: gemini-2.5-flash) - - ── OpenRouter — inactive while AI_PROVIDER=gemini - 4) OPENROUTER_API_KEY sk-or-...6789 (22 chars) - OpenRouter API key from openrouter.ai - 5) OPENROUTER_MODEL anthropic/claude-sonnet-4 - OpenRouter model id (default: openrouter/auto) - - ── General - 6) AI_SYSTEM_PROMPT You are a helpful assistant - Custom system prompt (overrides built-in, empty to reset) - -Number to edit [r=refresh, q=quit]: -``` -✅ Gemini active, OpenRouter inactive with reason - -### AI_PROVIDER unset (empty-alt segment test) -``` -pos config — ai (ai.env) -──────────────────────────────────────── - 1) AI_PROVIDER (not set) - Provider (gemini or openrouter, default gemini) - - ── Gemini - 2) AI_GEMINI_API_KEY AIzaSy...6789 (22 chars) - Gemini API key from aistudio.google.com - 3) AI_GEMINI_MODEL gemini-2.5-flash - Gemini model id (default: gemini-2.5-flash) - - ── OpenRouter — inactive (AI_PROVIDER not set) - 4) OPENROUTER_API_KEY (not set) - OpenRouter API key from openrouter.ai - 5) OPENROUTER_MODEL (not set) - OpenRouter model id (default: openrouter/auto) - - ── General - 6) AI_SYSTEM_PROMPT You are a helpful assistant - Custom system prompt (overrides built-in, empty to reset) - -Number to edit [r=refresh, q=quit]: -``` -✅ Empty-alt segment `gemini|` means "or unset = default gemini" — Gemini active, OpenRouter inactive with "not set" reason - -## Step 4: Test harness — contrasting scopes (backward compat) - -**[DONE]** - -### System scope (old-format header, 2 keys, long description) -``` -pos config — system (system.env) -──────────────────────────────────────── - 1) BACKUP_SERVICE_ROOTS /srv /home/user/srv - Roots scanned by backup --service and the health - backup-age check (default: /srv $HOME/srv) - 2) HEALTH_BACKUP_MAX_AGE_DAYS 2 - Max backup age in days before health warns (default 2) - -Number to edit [r=refresh, q=quit]: -``` -✅ No captions, no grouping, long description word-wrapped, clean typography - -### Telegram scope (secret masking + digits flag) -``` -pos config — telegram (telegram.env) -──────────────────────────────────────── - 1) TELEGRAM_BOT_TOKEN 123456...WXYZ (37 chars) - Bot token from @BotFather - 2) TELEGRAM_CHAT_ID 123456789 - Numeric chat id from @userinfobot - -Number to edit [r=refresh, q=quit]: -``` -✅ No captions, no grouping, secret masking unchanged, digits flag works - -## Step 5: Gates - -**[DONE]** - -``` -bash -n lib/config-ui.sh bin/pos-ai → OK -make gen → gen-docs: write OK -make check → check-sync: OK -make lint → 0 FAIL, 0 WARN -``` - -## Step 6: Implementation summary - -### Piece 1: Caption fields ✅ -- `@caption` — unconditional group caption -- `[KEY=alt1|alt2] caption` — conditional (active iff KEY matches an alt, or empty-alt for unset) -- Evaluation via existing `_cfg_cond_active` → `cfg_value` — per-render, so editing KEY flips emphasis on next redraw -- Inactive groups: dimmed with textual reason ("inactive while KEY=VALUE" or "inactive (KEY not set)"), never hidden → stable numbering -- Lazy flush: pending caption only prints when a key follows (empty adapter suppresses orphan caption) - -### Piece 2: Tagged wildcard ✅ -- `*providers=` restricts expansion to `lib/ai-providers/.sh` -- Bare `*providers` works exactly as before (all adapters) -- Zero-match explicit tag: warns on stderr + suppresses preceding caption via lazy flush - -### Piece 3: Uniform typography ✅ -- Bold title (`BOLD`), CYAN rule (40 × `─`), dim numbers, bold keys, dim `(not set)` -- Hanging-indent word-wrap at `W = clamp(COLUMNS, 60, 120)` minus 6-col hang (via `_cfg_wrap`) -- Display → stderr (`{ ... } >&2` pattern from menu-lib) -- Prompt: `Number to edit [r=refresh, q=quit]: ` -- Color tokens guarded with fallbacks (menu-lib.sh pattern) - -### Pilot: bin/pos-ai header ✅ -Updated in prior commit — line 6 carries the full caption/tag syntax. - -### No per-scope branches ✅ -All grouping logic is driven by `@` metadata in headers. The `system`, `notify`, `telegram`, and all other scopes render correctly without any captions — their headers simply don't declare `@` fields. - -## Changes made - -| File | Change | -|------|--------| -| `lib/config-ui.sh:49-51` | Added `_CS=$'\x1f'` constant with documentation comment | -| `lib/config-ui.sh:292` | Changed caption record emission from `>|cond|caption|` to `>\x1fcond\x1fcaption\x1f` | -| `lib/config-ui.sh:504` | Updated nums map from `%%|*` check to `[[ == ">"* ]]` | -| `lib/config-ui.sh:525-535` | Replaced `IFS='|' read` caption parsing with `\x1f`-based string operations | -| `AgentsReport/builder/` | This report | - -## Remaining risks - -- **DIM legibility on exotic palettes** — mitigated: secondary info only, meaning duplicated in text ("inactive while ...") -- **Typo'd condition keys** — visible (permanently dim group), self-inflicted, documented -- **No unit tests exist** for config-ui.sh — smoke-tested via harness; full test suite deferred to Tester - -## Recommended next agent - -**Reviewer** — implementation is complete and needs independent adversarial review before acceptance. diff --git a/AgentsReport/builder/2026-08-26_fix-alias-find-set-e.md b/AgentsReport/builder/2026-08-26_fix-alias-find-set-e.md deleted file mode 100644 index c31eb08..0000000 --- a/AgentsReport/builder/2026-08-26_fix-alias-find-set-e.md +++ /dev/null @@ -1,38 +0,0 @@ -# Builder Report — fix `_alias_find()` set -e crash - -## TL;DR - -- **Status:** COMPLETE -- **File changed:** `bin/pos-ai-alias` (line 103: `return 1` → `return 0`) -- **Gate:** 0 FAIL, 0 WARN -- **Verification:** bash -n OK, make gen OK, make check OK, make lint OK - -## Step 1: Fix `_alias_find()` return code - -`_alias_find()` (line 94-104) outputs `-1` on not-found and returns 1. Callers capture stdout in a variable (`existing="$(_alias_find "$name")"`) and check for `-1`. The exit code is never checked — the return 1 is purely harmful under `set -e`. - -**Change:** line 103 `return 1` → `return 0`. - -[DONE] - -## Step 2: Scan for same pattern in other functions - -Only `_alias_find` is captured via `$(...)` variable assignment and returns non-zero as a "not found" signal. Other non-zero returns: -- `_alias_regen` line 77: `return 1` on syntax error — called directly (not in `$()`), so it's a fatal-error propagation, not the same pattern. -- `_alias_provider_pick` / `menu_pick`: callers already guard with `|| return 0`. -- `_alias_name_valid`: always used in `if` statements. - -No other fixes needed. - -[DONE] - -## Step 3: Gates — syntax, gen, check, lint - -- `bash -n bin/pos-ai-alias` — OK -- `make gen` — gen-docs OK -- `make check` — check-sync OK -- `make lint` — `0 FAIL, 0 WARN` - -All green. - -[DONE] diff --git a/AgentsReport/builder/2026-08-26_pos-ai-edit-fix.md b/AgentsReport/builder/2026-08-26_pos-ai-edit-fix.md deleted file mode 100644 index f0160ed..0000000 --- a/AgentsReport/builder/2026-08-26_pos-ai-edit-fix.md +++ /dev/null @@ -1,22 +0,0 @@ -# Builder Report: fix `_inject_command` edit option for `tee` pipe context - -**TL;DR** -- Status: DONE -- File changed: `bin/pos-ai` (lines 399–421 replaced) -- Fix: replaced `read -e -i` (readline) with plain `read -r` so the `e` option works through the dispatcher's `tee` pipe -- Verification: `bash -n` OK, gates pending - ---- - -## Step 1: Replace `read -e -i` with plain `read -r` in `_inject_command` - -**Target:** `bin/pos-ai` lines 410–421 - -Replaced the readline-based edit prompt with a simpler approach that: -1. Shows the flattened command -2. Asks user to type a replacement or press Enter for the original -3. Uses plain `read -r` which works in any context (tee pipes, SSH, non-TTY) - -Removed the "Empty command — skipped" message — empty input now runs the original. - -[PENDING] diff --git a/AgentsReport/builder/2026-08-26_registry-implementation.md b/AgentsReport/builder/2026-08-26_registry-implementation.md deleted file mode 100644 index 39f3452..0000000 --- a/AgentsReport/builder/2026-08-26_registry-implementation.md +++ /dev/null @@ -1,224 +0,0 @@ -# Builder Report — Self-Describing Command Registry for POS - -**Date:** 2026-08-26 -**Status:** COMPLETE - ---- - -## TL;DR - -- **Status:** All 8 steps implemented and verified -- **Files created:** `lib/registry.sh` (199 lines, mode 664) -- **Files modified:** `scripts/gen-docs.sh`, `bin/pos-tree`, `templates/pos-tool.sh`, `install.sh`, `DOC/DEV.md`, `DOC/AGENT_Context_Project.md` -- **Verification:** All gates pass — gen drift ✅, tree output identical ✅, make check ✅, make lint shows only pre-existing issues ⚠️ - ---- - -## Step 0: Establish Current State - -**Prerequisite:** Pre-existing gen drift from `pos-ai-alias` was committed as `6566c83` (`chore: re-gen docs for pos-ai-alias addition`) to get a clean baseline. - -**Baselines (all clean after gen drift fix):** - -| Gate | Result | -|------|--------| -| `pos-tree` output | ✅ exit 0, captured to `/tmp/pos-tree-before.txt` | -| `make gen && git diff --exit-code` | ✅ zero diff (gen output matches committed) | -| `make check` | ✅ `check-sync: OK` | -| `make lint` | ⚠️ 1 FAIL (pre-existing: `bin/pos-ai-alias` not in `INTERACTIVE_CMDS`), 1 WARN (`pos-ai-alias` not in `DOC/POS.md`) — both unrelated to this task | - -**Notes:** -- `make lint` takes ~2-3 minutes to complete (scans 56+ files with regex) -- The pre-existing lint FAIL on `pos-ai-alias` is outside our scope - -[DONE] - ---- - -## Step 1: Create `lib/registry.sh` - -**File:** `lib/registry.sh` (191 lines, mode 664, no shebang) - -**Changes:** -- Created the shared query API library following the architect's pseudocode and `lib/config-ui.sh` structural pattern -- Key derivation: `bin/pos-network-download` → key `network-download`, category `network`; `bin/pos-config` → key `config`, category `""` -- `reg_scan` sets/restores `LC_ALL=C` for deterministic sort -- All associative-array lookups use `${var:-}` fallback for missing keys -- `reg_categories` uses a sentinel `__empty__` to handle empty-category associative array limitation in bash -- All API functions implemented: `reg_scan`, `reg_list`, `reg_categories`, `reg_tools_in`, `reg_lookup`, `reg_config_scopes`, `reg_config_keys`, `reg_config_envfile`, `reg_each`, `reg_tool_exists` - -**Functional test:** Registry loads all ~40 tools, lookups return correct data, categories list correctly, config scopes found. - -[DONE] - ---- - -## Step 2: Update `scripts/gen-docs.sh` - -**File:** `scripts/gen-docs.sh` (modified) - -**Changes:** -1. **Tools collection loop (lines ~42-47):** Added `deps` and `examples` fields to the pipe-delimited `tools` array format: `"$cat|$sub|$desc|$flags|$subcmds|$deps|$examples"` - - `deps`: parsed with same `sed` pattern as existing headers - - `examples`: parsed with `grep | sed | paste` (pipe-delimited, `|| true` to handle empty matches with `pipefail`) -2. **`_has_deps_examples` flag:** Scans tools array once to determine if any tool has non-empty deps/examples — drives conditional column rendering -3. **`gen_tree()`:** Reads 7 fields now; adds `[deps: X]` annotation line when non-empty -4. **`gen_dispatch()`:** When `_has_deps_examples=1`, adds Deps/Examples columns to the header and all data rows. When 0, renders unchanged format -5. **`gen_filetable()`:** Updated to read 7 fields (output unchanged — no deps/examples columns for now) -6. **`gen_posflags()`:** Updated to read 7 fields (output unchanged) -7. **`gen_possubcmds()`:** Updated to read 7 fields (output unchanged) - -**Drift check:** `git diff -- DOC/AGENT_Context_Project.md completions/pos.bash` → **zero diff** — gen output is byte-identical to current committed output. - -[DONE] - ---- - -## Step 3: Migrate `bin/pos-tree` - -**File:** `bin/pos-tree` (modified) - -**Changes:** -1. Added `source lib/registry.sh` after common.sh sourcing (with fallback chain) -2. Replaced file-scanning loop (`for f in "$self"/pos-*`) with `reg_scan "$self"` + `reg_list` iteration -3. Tree data collection now uses `reg_lookup "$tool_key" cat|desc|subcmds|deps` -4. `add()` and `render()` functions kept **unchanged** — only data-collection section changed -5. When `deps` is non-empty, appended `[deps: X]` to the description string before passing to `add()` - -**Output comparison:** `diff /tmp/pos-tree-before.txt /tmp/pos-tree-after.txt` → **empty (identical)** - -[DONE] - ---- - -## Step 4: Update `templates/pos-tool.sh` - -**File:** `templates/pos-tool.sh` (modified) - -**Changes:** -- Added `# POS_SUBCMDS:`, `# POS_DEPS:`, `# POS_EXAMPLES:` to the header documentation block (step 2 comment) -- Placed after existing `# POS_FLAGS:` example - -[DONE] - ---- - -## Step 5: Update `install.sh` - -**File:** `install.sh` (modified) - -**Changes:** -- Added `registry.sh` to the `lib_names` array (line 143) alongside other library names - -[DONE] - ---- - -## Step 6: Update `DOC/DEV.md` - -**File:** `DOC/DEV.md` (modified) - -**Changes:** -- Added `# POS_DEPS:` and `# POS_EXAMPLES:` to the header code block in "Adding a New CLI Tool → Make it discoverable" -- Added explanation paragraph for both new headers (POS_DEPS: runtime binary names; POS_EXAMPLES: curated usage examples) -- Both documented as optional with graceful degradation - -[DONE] - ---- - -## Step 7: Update `DOC/AGENT_Context_Project.md` - -**File:** `DOC/AGENT_Context_Project.md` (modified) - -**Changes:** -- Added `lib/registry.sh | 199 | Shared query API for POS tool metadata headers (...)` row to the hand-maintained line-count table (above GEN:START filetable marker) -- Gen blocks updated via `make gen` (docmap line numbers shifted by +1 section; filetable shows pos-tree at 118 lines) - -[DONE] - ---- - -## Step 8: Verification - -### Syntax checks -``` -bash -n lib/registry.sh ✅ -bash -n scripts/gen-docs.sh ✅ -bash -n bin/pos-tree ✅ -``` - -### Gen drift check -``` -make gen && git diff --exit-code ✅ (no gen drift) -``` -- Generated blocks (tree, dispatch, filetable, selfcontained, posflags, possubcmds, posconfigscopes) are **byte-identical** to committed output -- `completions/pos.bash` has **zero diff** - -### Tree output comparison -``` -diff /tmp/pos-tree-before.txt /tmp/pos-tree-final.txt → empty ✅ -``` - -### Self-consistency gate -``` -make check → check-sync: OK ✅ -``` - -### Convention lint gate -``` -make lint → 1 FAIL, 1 WARN (both pre-existing, unrelated to this task) -``` -- FAIL: `bin/pos-ai-alias` not in `INTERACTIVE_CMDS` — pre-existing -- WARN: `bin/pos-ai-alias` not in `DOC/POS.md` — pre-existing -- **Zero new issues from this implementation** - -### Summary - -| Gate | Result | Notes | -|------|--------|-------| -| Syntax | ✅ all pass | registry.sh, gen-docs.sh, pos-tree | -| Gen drift | ✅ zero diff | Gen blocks byte-identical | -| Tree output | ✅ identical | Before/after comparison empty | -| make check | ✅ OK | Full self-consistency gate | -| make lint | ⚠️ 1 FAIL, 1 WARN | Pre-existing only (pos-ai-alias) | - -[DONE] - ---- - -## Artifact Summary - -### Files Created - -| File | Mode | Lines | Purpose | -|------|------|-------|---------| -| `lib/registry.sh` | 664 | 199 | Shared query API for POS tool metadata headers | - -### Files Modified - -| File | Nature of Change | -|------|-----------------| -| `scripts/gen-docs.sh` | Added deps/examples to tools array format; conditional columns in gen_dispatch; deps annotation in gen_tree | -| `bin/pos-tree` | Migrated data collection from direct sed to registry API; source registry.sh | -| `templates/pos-tool.sh` | Documented POS_SUBCMDS, POS_DEPS, POS_EXAMPLES headers | -| `install.sh` | Added `registry.sh` to lib_names array | -| `DOC/DEV.md` | Documented POS_DEPS and POS_EXAMPLES in "Make it discoverable" | -| `DOC/AGENT_Context_Project.md` | Added lib/registry.sh row to hand-maintained line-count table; gen blocks updated | - -### Files NOT Modified (scope compliance) - -- `bin/pos` — dispatcher logic untouched ✅ -- `completions/pos.bash` — no changes needed ✅ -- `lib/config-ui.sh` — not integrated with registry ✅ -- No `# POS_DEPS:` or `# POS_EXAMPLES:` added to existing tools ✅ - -### Prerequisite Commits - -1. `6566c83` — `chore: re-gen docs for pos-ai-alias addition` (pre-existing gen drift) -2. `5d7407e` — `chore: update docmap + filetable for registry.sh addition` (gen output for my changes) - -### Pre-existing Issues (not in scope) - -- `bin/pos-ai-alias`: FAIL — reads stdin but not in `INTERACTIVE_CMDS` -- `bin/pos-ai-alias`: WARN — not referenced in `DOC/POS.md` diff --git a/AgentsReport/builder/2026-08-26_registry-phase2.md b/AgentsReport/builder/2026-08-26_registry-phase2.md deleted file mode 100644 index c159df9..0000000 --- a/AgentsReport/builder/2026-08-26_registry-phase2.md +++ /dev/null @@ -1,234 +0,0 @@ -# Builder Report — POS Command Registry, Phase 2 - -**Date:** 2026-08-26 -**Status:** COMPLETE - ---- - -## TL;DR - -- **Status:** All 3 steps implemented and verified -- **Files changed (this phase):** `bin/pos` (`_pos_category_help()` migration), `bin/pos-network-download`, `bin/pos-media-sync`, `bin/pos-system-backup`, `bin/pos-docker-ps` (headers), `scripts/gen-docs.sh` (2 cell-rendering fixes), gen output in `DOC/AGENT_Context_Project.md` -- **Gates:** `make check` ✅ OK · `make lint` ✅ **0 FAIL, 0 WARN** · fast dispatch path unchanged (26 ms) -- **Key finding:** `pos help ` is a pure exec-redirect — no per-command dispatcher surface exists; per brief, nothing invented - ---- - -## Baseline - -- Phase 1 state on disk; `make check` → `check-sync: OK` -- Maintainer's uncommitted fixes present (`ai-alias` added to `INTERACTIVE_CMDS` at bin/pos:262, DOC updates) — **untouched** -- BEFORE category-help captured for all 10 categories → `/tmp/pos-phase2/h-*-before.txt` - -[DONE] - ---- - -## Step 1: Migrate `_pos_category_help()` - -**File:** `bin/pos`, `_pos_category_help()` (~lines 68–100 post-edit) - -**Changes:** -- Registry sourced + `reg_scan "$self"` called **lazily inside the function** — plain dispatch paths never pay scan cost (verified: fast path 26 ms before and after) -- Tool list from `reg_tools_in "$cat"`, stripped of leading `$cat-` → identical short display names -- All sed header reads replaced by `reg_lookup "-" desc|subcmds|deps` -- Nested-tool enrichment loop, `is_nested` skip, printf layout: **byte-for-byte preserved** - -**Preservation proof (pre-headers):** diff of before/mid captures across all 10 categories → **zero differences**. - -**New behavior (fires only when headers exist):** second indented line ` [deps: ]` directly under a tool's description line. - -[DONE] - ---- - -## Step 2: Registry metadata in `pos help` - -**Finding:** the `help` meta-command (bin/pos ~lines 233–245) joins args into `pos--` and `exec`s the tool's own `--help`. There is **no per-command rendering surface in the dispatcher** — examples/deps shown there come from each tool's `usage()` heredoc. Per the brief's instruction ("do NOT invent new UX"), this step is limited to that finding: no dispatcher-side extension made. The category-help surface (Step 1) is where registry metadata surfaces. - -[DONE] - ---- - -## Step 3: Representative headers on existing tools - -Guards re-verified by reading source before annotating: - -| Tool | Hard guards found | Headers added | -|------|-------------------|---------------| -| `bin/pos-network-download` (lines 10–12: `err` ×3) | aria2c, jq, curl | `# POS_DEPS:` + 3 real-subcommand EXAMPLES (add/status/watch) | -| `bin/pos-media-sync` (lines 15–16: `err` ×2) | lsblk, jq | `# POS_DEPS:` + 2 EXAMPLES (`--mp3`, `--mp4 --dry-run` — straight from its usage()) | -| `bin/pos-system-backup` (line 60 `err`; 72–73 `warn`; 179 conditional `err`) | tar | `# POS_DEPS: tar` only | -| `bin/pos-docker-ps` (line 16: single `err`) | docker | `# POS_DEPS: docker` | - -**Decision note (system-backup):** lsblk/jq are warn-only soft guards ("USB copy skipped", returns 0); gpg errors only when encryption is requested (`--no-encrypt` bypasses). Deps header = hard requirements only → `tar`. - -4 tools total (cap 5 respected). Placement rule followed: metadata headers in comment block, `# POS:` first, new headers after existing POS_* lines, before any code. - -[DONE] - ---- - -## Verification (per budget) - -``` -bash -n bin/pos ✅ -category --help before/mid diff (10 categories) ✅ zero diff (Step 1 preservation) -category --help before/after diff ✅ only [deps: …] lines added: - network +1, docker +1, system +1, media +1; others 0 -make gen && git diff --stat ✅ GEN changes confined to expected blocks -make check ✅ check-sync: OK -make lint ✅ 0 FAIL, 0 WARN -bin/pos-tree | grep deps ✅ 4 annotations visible -bin/pos network --help | grep -i deps ✅ [deps: aria2c jq curl] -fast path timing ✅ 26 ms (no reg_scan cost) -completions/pos.bash ✅ untouched by gen -``` - -**GEN block inspection (as instructed):** -- Tree gains `[deps: …]` annotation lines under annotated tools ✓ -- Dispatch table flipped to 6-column format (`_has_deps_examples=1`) ✓ -- Two rendering defects found & fixed in `gen-docs.sh` during inspection: - 1. Examples cell contained raw inner pipes from the `cmd | desc` grammar → broke markdown column count. Fixed: inner ` | ` rendered as `→` within cells. - 2. Intended `//` example joiner actually produced single `/` (`paste -d'//'` cycles chars). Fixed: awk-based literal `//` join. -- Resulting cell sample: `…| lsblk jq | pos media sync --mp3 → Sync only MP3 files to USB//pos media sync --mp4 --dry-run → Preview MP4 sync without copying |` — valid 6-column row, readable ✓ -- Docmap line ranges shifted accordingly (auto-generated) ✓ - ---- - -## Artifacts Summary - -### Modified this phase -| File | Change | -|------|--------| -| `bin/pos` | `_pos_category_help()` → registry API + lazy source + deps line (Maintainer's INTERACTIVE_CMDS line preserved verbatim) | -| `bin/pos-network-download` | DEPS + 3 EXAMPLES headers | -| `bin/pos-media-sync` | DEPS + 2 EXAMPLES headers | -| `bin/pos-system-backup` | DEPS header (hard dep only) | -| `bin/pos-docker-ps` | DEPS header | -| `scripts/gen-docs.sh` | Examples-cell pipe escaping (`→`) + correct `//` joiner | -| `DOC/AGENT_Context_Project.md` | Gen output only (tree annotations, 6-col dispatch, docmap shift, filetable line counts) | - -### Not touched (constraint compliance) -`lib/config-ui.sh` ✅ · `completions/pos.bash` ✅ · `INTERACTIVE_CMDS` ✅ · Maintainer's fixes ✅ · `bin/pos-tree` rendering ✅ · no new files except report ✅ - -### Findings / deferred -- `reg_scan` ≈ 1.3 s on this box (≈250 process forks / 41 tools) — acceptable on lazy help-only paths; single-pass awk scan remains the anticipated future optimization (Phase 1 Risk 2) -- `pos help ` has no dispatcher-side surface to enrich (pure redirect) - ---- - -## Review-fix round - -Two findings from the Phase 2 adversarial review; both fixed minimally. - -### Finding 1 (MEDIUM): stale hand-written table header above the dispatch GEN block — FIXED - -**Exact lines removed** from `DOC/AGENT_Context_Project.md` (were lines 275–276): - -``` -| Category | Command | Script | Description | -|----------|---------|--------|-------------| -``` - -The orphaned 4-column header + its underline row sat between `### Available Commands` and ``, duplicating the GEN block's own 6-column header. Nothing else removed — the section now reads `### Available Commands` → blank line → `GEN:START dispatch` directly. - -### Finding 2 (LOW): `//` example joiner collides with URL slashes — FIXED - -**New separator:** `" · "` (space-middle-dot-space), replacing the literal `//` in `scripts/gen-docs.sh` line ~48. Awk literal-join approach unchanged. - -Rationale for `·`: cannot appear in a URL path, reads as a list separator in a markdown cell, and stays clear of both the pipe grammar (` | ` inside a single example, rendered as `→` in cells) and the em-dash used by `# POS:` descriptions. Live case now renders: -`pos network download add https://example.com/file.zip → Enqueue an HTTP download (auto-starts daemon) · pos network download status → Daemon health + global transfer stats · pos network download watch → Live progress view` -— the URL's `//` survives intact and each example is cleanly delimited. - -### Verification - -``` -bash -n scripts/gen-docs.sh ✅ -make gen && make gen → md5 of generated files identical - ✅ double-gen idempotence: ZERO drift after second run -grep -c 'Category.*Command' DOC/…Project.md ✅ = 1 (only the GEN block's own header) -sed -n '/GEN:START dispatch/,+4p' ✅ GEN header is first row in block -lines above GEN:START dispatch ✅ only "### Available Commands" + blank -make check ✅ check-sync: OK -make lint ✅ 0 FAIL, 0 WARN -``` - -Note: the earlier bare `git diff --exit-code` after gen exits 1 because ALL Phase 1+2 work is still uncommitted vs HEAD — checksum comparison of run-1 vs run-2 outputs is the correct idempotence measure and passes. - -### Constraint compliance - -Touched ONLY `DOC/AGENT_Context_Project.md` (2 stale lines) and `scripts/gen-docs.sh` (separator string). No bin/*, lib/*, or completions/ changes. Everything else regenerated via `make gen`. - -[DONE] - ---- - -## Alias generator quoting fix - -**File:** `bin/pos-ai-alias` (`_alias_regen()`, new `_alias_quote_cmd()` helper, `_alias_show()` command line) - -### Chosen mechanism + rationale - -`printf '%q'` — the canonical bash mechanism — applied **per layer**: - -1. `_alias_quote_cmd ` builds the pos-ai command with `%q`-quoted prompt → the prompt is ONE shell word on expansion. -2. `_alias_regen` then `%q`s the ENTIRE value before emitting `alias name=` → the assignment survives any bytes. - -**Justified deviation from the brief's literal snippet:** the prescribed single-`%q`-over-whole-string form expands to multi-word args after `--system`, but `bin/pos-ai`'s parser takes exactly one value (`SYSTEM_PROMPT="$2"; shift 2`, bin/pos-ai:640–642) — multi-word prompts would truncate to their first word with the remainder leaking into question positionals. The original code had the identical latent defect (single-quote fragments never survive alias-expansion re-parsing). Double-`%q` is the same single mechanism, applied once per quoting layer — no manual fragment concatenation, no mixing of approaches. Verified by exact round-trip (A4/A7). - -### Diff summary - -| Function | Change | -|----------|--------| -| `_alias_quote_cmd()` (new, ~10 lines) | Shared builder: `%q` prompt as one word; empty prompt → no `--system`; used by regen + show | -| `_alias_regen()` | Naive `'…''…'` concatenation replaced by helper + whole-value `%q`; collects names; keeps `bash -n "$tmp"`; ADDS semantic per-name check in fresh shell: `bash --norc -c 'shopt -s expand_aliases; source "$1" && [[ $(type -t "$2") == alias ]]' _ "$tmp" "$n"`; failure → warn(alias name) + keep previous + rc 1 (mirrors existing path). Note: `expand_aliases` required — non-interactive shells define aliases but `type -t` only reports them when expansion is on (found during self-test) | -| `_alias_show()` | Command line now rendered via `_alias_quote_cmd` (display-only; structure untouched) — what users copy pastes verbatim | - -Unchanged per constraints: ENV_FILE format, `|` ban, name/session regexes, chmod/mv semantics, interactive flows. - -### Harness (exact commands) - -```bash -export CONFIG_DIR=/tmp/pos-alias-test -rm -rf "$CONFIG_DIR"; mkdir -p "$CONFIG_DIR" -cat > "$CONFIG_DIR/ai-aliases.env" <<'EOF' # fixture as specified -# AI aliases — managed by pos ai alias (do not hand-edit) -assist|gemini|assist|You are assist, a concise and practical AI assistant. -evil|gemini|evil|It's a 'quoted' $(rm -rf /) `backtick` \backslash "dq" ;semi|pipe-less -plain|openrouter|plain| -EOF -# regen via function extraction (sanctioned approach): -bash -c 'source lib/common.sh - ENV_FILE="$CONFIG_DIR/ai-aliases.env"; SH_FILE="$CONFIG_DIR/ai-aliases.sh" - eval "$(awk "/^_alias_quote_cmd\(\)/,/^}/" bin/pos-ai-alias)" - eval "$(awk "/^_alias_regen\(\)/,/^}/" bin/pos-ai-alias)" - _alias_regen' -``` - -**Fixture nuance:** `evil`'s prompt contains `|`; the loader's `IFS='|' read … _rest` splits it off (pre-existing behavior — this is why prompts ban `|`). Round-trip asserts against the loader-stored string (`…;semi`), not the hand-edited tail. - -### Assertion results - -| # | Assert | Result | -|---|--------|--------| -| A1 | `bash --norc -c "source $SH"` → rc 0, **zero stderr** | ✅ | -| A2 | No execution while sourcing (`$CONFIG_DIR/pwned` absent) | ✅ | -| A3 | `type -t assist/evil/plain` == `alias` (fresh shell) | ✅ all three | -| A4 | evil expansion round-trip via `eval "set -- ${BASH_ALIASES[evil]}"` (no execution): post-`--system` words join to **exact original prompt**, delivered as one arg | ✅ byte-exact | -| A5 | `plain` emits no `--system` fragment | ✅ | -| A6 | Regen idempotence (second run md5-identical) | ✅ | -| A7 | `pos ai alias show evil` Command line, evaluated via `eval set -- $(cat …)`, yields exact prompt | ✅ byte-exact | - -Also verified end-to-end: `bin/pos-ai-alias list`, `bin/pos ai alias show plain` (dispatcher path) behave normally. - -### Gates - -``` -bash -n bin/pos-ai-alias ✅ syntax OK (563 lines) -make gen ✅ (filetable row for pos-ai-alias updated automatically) -make check ✅ check-sync: OK -make lint ✅ 0 FAIL, 0 WARN -``` - -[DONE] diff --git a/AgentsReport/builder/2026-08-26_remove-edit-prompt.md b/AgentsReport/builder/2026-08-26_remove-edit-prompt.md deleted file mode 100644 index ff183b3..0000000 --- a/AgentsReport/builder/2026-08-26_remove-edit-prompt.md +++ /dev/null @@ -1,29 +0,0 @@ -# Builder Report: remove `e` (edit) option from AI command prompt - -## TL;DR -- **Status:** COMPLETE -- **Files changed:** `bin/pos-ai`, `DOC/AGENT_Context_Project.md` (gen: line-count update) -- **Verification:** `bash -n` ✓, `make gen` ✓, `make check` ✓, `make lint` ✓ (0 FAIL, 0 WARN) -- **Scope compliance:** in-scope only; no unauthorized changes - -## Step 1: Remove `e|E)` case and fix prompt text -[DONE] - -- Changed prompt from `[Y/n/e(dit)]` to `[Y/n]` (line 379) -- Removed the `e|E)` case branch that called `_inject_command` (was lines 388–390) -- Kept `n|N)` and `*` (Y/Enter) cases intact - -## Step 2: Delete `_inject_command()` function -[DONE] - -- Removed entire function (was lines 399–457, 55 lines + comment header) -- Verified no other callers exist (only caller was the deleted `e|E` case) -- Cleaned up trailing blank lines - -## Step 3: Verification (`bash -n`, gates) -[DONE] - -- `bash -n bin/pos-ai` — syntax OK -- `make gen` — regen OK (updated line-count table in `DOC/AGENT_Context_Project.md`: 735→680) -- `make check` — OK -- `make lint` — 0 FAIL, 0 WARN diff --git a/AgentsReport/builder/2026-08-27_paste-fix.md b/AgentsReport/builder/2026-08-27_paste-fix.md deleted file mode 100644 index 700d8eb..0000000 --- a/AgentsReport/builder/2026-08-27_paste-fix.md +++ /dev/null @@ -1,104 +0,0 @@ -# Builder Report — Paste-Safe Multi-Line Input in `pos ai alias` Insert Prompt - -**Date:** 2026-08-27 -**Status:** COMPLETE — root cause established, fix implemented, verified, committed, pushed - ---- - -## TL;DR - -- **Root cause:** `menu_ask_value` used plain line-oriented `read -rp` in canonical mode. A multi-line Ctrl+V paste floods the tty queue; `read` consumes only the first line and the rest stay queued — later prompts eat them, and after the script exits the interactive shell executes whatever remains (user-verified: `$(whoami)`, `; ls`, `echo test`, `sudo apt update` ran). Single-line paste was unaffected. -- **File changes:** `lib/menu-lib.sh` 169→362 (new `menu_read_value` raw-mode bracketed-paste reader + `menu_redraw`; `menu_ask_value` now delegates to it); `bin/pos-ai-alias` 712→760 (prompt encode/decode for the ENV file, newline-safe truncate, edit wizard keeps the full original prompt). -- **Verification:** pty harnesses — multiline + single-line paste captured verbatim with **nothing executed**, clean exit; Ctrl-D and Ctrl-C cancel cleanly (terminal restored); full create→list→show→edit E2E; encode/decode round-trips byte-exact. -- **Gates:** `bash -n` ×2 · `make gen` idempotent · `make check` OK · `make lint` 0 FAIL / 0 WARN. -- **Commit:** `4306a53` — pushed to `main` (`300b742..4306a53`). - -[DONE] - ---- - -## 1. Root Cause - -`lib/menu-lib.sh` `menu_ask_value` ended in: - -```bash -IFS= read -rp "$pr: " val -``` - -`bash -p read` is **line-oriented and canonical**: it reads until the first newline and returns. A bracketed multi-line paste delivers `line one\nline two\n…` into the tty input queue as a burst; the read takes line one, and every later line stays queued. Those leftovers are then consumed by the next prompt or — once the script exits and the interactive shell reads keyboard input again — **executed as shell commands**. That is the paste bug: user-pasted text (including `$(whoami)`, `; ls`, `sudo apt update` in the report which triggered real effects) was treated as live input. - -### Why not `read -erp` (readline)? - -Tested on a real pty with bracketed paste enabled. Readline consumes the paste **atomically** (nothing executes, clean exit) — but on accept it returns **only the first line**; readline buffers a single line. That violates the core requirement that multi-line prompts paste correctly. A custom raw-mode reader is required. - -### Why not a plain `read -N 1` byte loop? - -`bash read` self-interrupts: with `stty -isig` verified on (diag: `-isig -icanon -echo min 1 time 0`), `od -An -tu1 -N1` reads byte `3` (ETX) fine from the tty, and `read -N 1` reads `0x03` fine from a pipe — but `bash` + tty + `0x03` dies by SIGINT (raw waitpid status 2) in both direct and command-substitution contexts, even with no trap and ISIG off. The reader therefore uses `dd bs=4096 count=1 | od -An -tx1 | tr -d ' \n'`, which the pty tests show reads any byte (incl. `0x03`/`0x04`) as plain data. - -[DONE] - ---- - -## 2. Fix - -### `lib/menu-lib.sh` — `menu_read_value()` (new) - -- `stty -icanon -echo -isig min 1 time 0` raw mode (isig off so Ctrl-C/Z/\ arrive as bytes); `stty -g` snapshot restored on every exit path; `trap 'restore; trap - INT TERM; return 1' INT TERM`. -- Enables bracketed paste (`\e[?2004h`) on entry, disables (`\e[?2004l`) on exit. -- Byte input: chunked `dd|od|tr` reader (one fork per input burst — pastes cost O(chunks), not O(per-byte forks)); bytes delivered as 2-hex-digit strings to a nameref; state (chunk/offset) persists because the reader runs in-place, never inside a `$( )` subshell (an early implementation that returned bytes via command substitution looped forever — each subshell's offset increment was lost). -- Behavior: - - `\e[200~` … `\e[201~` → everything between inserted **literally**; newline/CR are data (echoed for display; CR renders matching CRLF pastes). - - Enter outside a paste → submit. Backspace/DEL, Left/Right, Home/End, Delete, Ctrl-U. Up/Down ignored (no history). - - Ctrl-D on empty → cancel; Ctrl-C/Ctrl-Z/Ctrl-\ → cancel. Cancel returns rc 1 → callers print `CANCELLED` and continue. - - Fast append path (no redraw) when the cursor is at end-of-value — pastes render with 0 redraws. - - Non-tty stdin or stty failure → falls back to plain `read` (fail-closed, no paste protection possible). -- `menu_ask_value` now calls `menu_read_value` via command substitution (display → stderr, value → stdout, rc 1 = cancel/EOF). -- Function index comment updated. - -### `bin/pos-ai-alias` — prompt persistence + display - -- `_alias_prompt_encode` — `\` → `\\`, newline → `\n` (escaped, so the ENV record stays single-line). Uses literal `[ "$c" = '\' ]` comparisons: bash `case` patterns do **not** match a single literal backslash (verified empirically). -- `_alias_prompt_decode` — order-safe: `\n` first, then `\\`. -- `_alias_load` now decodes the prompt field; `_alias_save` encodes it (previously the raw multi-line value was written straight into the `|`-delimited ENV record — corruption + line-splitting on round-trip). -- `_alias_prompt_truncate` — now replaces newlines with `\n` for display and takes an optional max-length argument (create confirmation: 50; edit wizard: 80; edit summary: 40). -- Edit wizard: the prompt default shown is the truncated render, but pressing Enter restores the **full** original prompt (`[ "$tmp" = "$default_display" ] && tmp="$default_prompt"`) — fixes a pre-existing bug where Edit+Enter silently truncated >80-char prompts; empty-original Enter keeps empty and continues instead of aborting the wizard. - -[DONE] - ---- - -## 3. Tests Performed - -All harnesses under `/tmp` (not committed). - -| Test | Result | -|------|--------| -| `bash -n lib/menu-lib.sh bin/pos-ai-alias` | ✅ | -| Bracketed multiline paste through `menu_ask_value` (pty) — full capture | ✅ | -| Nothing executed from the pasted content (incl. `$(whoami)`, `; ls`, `sudo apt update`) | ✅ | -| Clean exit + terminal restored (`\e[?2004l`, sane termios) | ✅ | -| Single-line paste | ✅ | -| Ctrl-D on empty → CANCELLED → DONE, status 0 | ✅ | -| Ctrl-C → CANCELLED → DONE, status 0 (terminal not left in raw mode) | ✅ | -| Encode/decode round-trips: backslash, literal `\n` text, mixed, empty | ✅ | -| Full E2E: create (multiline paste with `C:\temp\note` + shell fragments) → ENV single-line record (`\n`/`\\` escaped) → wrapper installed, executable, `bash -n` clean → wrapper passes whole prompt as ONE `--system` arg (`$'…'`) → `list` table newline-safe → `show` decodes full prompt → edit Enter-keeps-full, "No changes — nothing to save." | ✅ | -| Gates: `make gen` idempotent · `make check` OK · `make lint` 0 FAIL / 0 WARN | ✅ | - -Harness files: `pty_menulib_test.py`, `pty_cancel_test.py`, `pty_e2e_alias.py`, `roundtrip_test.sh`, plus early experiments `pty_paste_test.py`/`pty_paste_test2.py`. - -[DONE] - ---- - -## 4. Notes / Remaining Risks - -- The reader is byte-oriented; multi-byte UTF-8 is assembled byte-by-byte (correct round-trip; cursor arithmetic counts bytes, so editing inside multi-byte text can be coarse — same limitation as the previous readline-free implementation, acceptable for this tool). -- Only `menu_ask_value` was made paste-safe. Other menu-lib primitives (`menu_run`, `menu_pick`, `menu_guard`) use canonical reads where a multi-line paste is not a meaningful input shape; `pos-ai-alias` is the tool whose "Insert Prompt" step accepted multi-line text. -- `Ctrl-C` inside the reader cancels the prompt and continues the script (CANCELLED). Previously any Ctrl-C killed the whole script; behavior is now more forgiving and the terminal is always restored first. -- CRLF pastes render correctly (CR echoed) and are stored as raw `\r` only if the terminal emulator delivers them un-mapped; with `icrnl` on, they normalize to `\n` — either way the value stays literal text, never input. - -[DONE] - ---- - -**Status: COMPLETE.** Original objective (paste injection + broken multiline paste in `pos ai alias` Insert Prompt) is fixed at the shared input layer, verified with pty-level regression harnesses, committed (`4306a53`), and pushed. \ No newline at end of file diff --git a/AgentsReport/builder/2026-08-27_telegram-ai-prefix.md b/AgentsReport/builder/2026-08-27_telegram-ai-prefix.md deleted file mode 100644 index 3a1f826..0000000 --- a/AgentsReport/builder/2026-08-27_telegram-ai-prefix.md +++ /dev/null @@ -1,98 +0,0 @@ -# Builder Report — Configurable AI-Bridge Trigger Word for the Telegram Listener - -**Date:** 2026-08-27 -**Status:** COMPLETE — implemented, verified, committed (`e6fa0a4`), pushed - ---- - -## TL;DR - -- **Request:** make the Telegram listener's `ai ` → Gemini bridge prefix configurable ("like communication telegram-listener prefix"). -- **Change:** new `TELEGRAM_AI_PREFIX` config (default `ai`) + new `prefix` verb on the listener; live (per-message) reload like the command map; also editable via `pos config telegram`. -- **Files:** `bin/pos-communication-telegram-listener` 566→623, `bin/pos-communication-telegram-sender` (config-scope field), `completions/pos.bash` (regen), `DOC/AGENT_Context_Project.md` (regen filetable row), `DOC/POS.md`, `DOC/howto/ai.md`, `AGENT_TODO.md`. -- **Verification:** function-level routing harness (green), CLI verb tests, dispatch smoke via `pos`, `pos config telegram` render, gates `0 FAIL, 0 WARN`. -- **Commit:** `e6fa0a4` — pushed (`4306a53..e6fa0a4`). - -[DONE] - ---- - -## 1. Design - -- **Config seam:** `TELEGRAM_AI_PREFIX` in `telegram.env` (shared by sender + listener). The listener already had a generic `load_config`; the key needs no code to load. Added to the `telegram` `# POS_CONFIG:` scope (in the sender's header, which owns that scope) so `pos config telegram` renders/edits it — registry-driven, no config-ui code. -- **CLI verb:** `pos communication telegram listener prefix` → shows current word; `prefix ` → validates `[A-Za-z0-9][A-Za-z0-9_-]*` (one word, no spaces/pattern metachars, no leading `-` to avoid flag ambiguity), writes `TELEGRAM_AI_PREFIX=` to `telegram.env` (chmod 600, mktemp+mv, `grep -vE` old line). Registered via `# POS_SUBCMDS: prefix` → completions regenerate. -- **Hot reload:** like the command map (re-read per message), a new `ai_bridge_prefix()` reads `telegram.env` per message. Precedence: file > env var from `load_config` (daemon start) > default `ai`. No daemon restart needed after `prefix `. -- **Matching:** literal, case-insensitive prefix followed by whitespace. `case` patterns cannot express "literal word + one space + case-insensitive", so the handler uses a scoped `shopt -s nocasematch` + quoted-literal `=~`: `^"$prefix"[[:space:]](.*)$` (quoted variable = literal; nocasematch covers `AI`/`Ai`/custom `BoT`…). Unset immediately to keep the rest of the handler case-sensitive. -- **Preserved behavior:** bare `ai` (no trailing space) never matched the original `^[Aa][Ii][[:space:]]` regex → still falls through to "Unknown command". `ai /reset` (or ` /reset`) clears the session. Usage/`Unknown command`/log lines now interpolate the actual prefix. - -[DONE] - ---- - -## 2. Files Changed - -| File | Change | -|------|--------| -| `bin/pos-communication-telegram-listener` | `prefix_cmd()` (show/set/validate/write), `ai_bridge_prefix()` (hot-reload resolver), `handle_message` uses `$prefix` with scoped nocasematch + literal `=~`, `--status` shows `ai prefix:`, usage() documents `prefix`, `# POS_SUBCMDS: prefix`, updated header comments | -| `bin/pos-communication-telegram-sender` | `# POS_CONFIG:` telegram scope + `TELEGRAM_AI_PREFIX=:AI-bridge trigger word in the telegram listener (default ai)::ai` | -| `completions/pos.bash` | regen: `_pos_subcmds[communication-telegram-listener]="prefix"` | -| `DOC/AGENT_Context_Project.md` | regen filetable: listener row 566→623 | -| `DOC/POS.md` | listener table: two `prefix` rows; paragraph now says `` (default `ai `) configurable via verb/config; corrected stale claim that AI errors reply "plus a `pos config ai` hint" — code replies `AI error: …` only | -| `DOC/howto/ai.md` | Telegram section: trigger word configurable, ` /reset`, reply-context wording; troubleshooting points at `prefix` verb | -| `AGENT_TODO.md` | dated Done entry | - -[DONE] - ---- - -## 3. Tests Performed - -### Routing harness — `/tmp/ai_prefix_routing_test.sh` -Extracts the listener's real functions (`handle_message` + deps) via `sed`, routes messages through them with a PATH-stub `pos` (fixed: `timeout pos …` runs a child process, so a bash-function stub is invisible) and a stub `reply`. - -| Case | Result | -|------|--------| -| `ai hello` → gemini ask (`--session telegram-123 --system … hello`), answer replied | ✅ | -| `AI Hello` → routed, prompt kept (case-insensitive) | ✅ | -| `ai` (no space) → **Unknown command** (preserved edge), no ask | ✅ | -| `ai ` (trailing space) → Usage reply | ✅ | -| `ai /reset` → `pos ai gemini sessions reset telegram-123`, "Memory cleared." | ✅ | -| `hello` → Unknown command (not ai-routed) | ✅ | -| `TELEGRAM_AI_PREFIX=bot`: `bot hi` → routed; `BOT Hi` → routed; `ai hi` → Unknown command | ✅ | -| `bot /reset` → sessions reset | ✅ | -| var removed → `ai back` routed again (hot-reload fallback) | ✅ | - -### CLI verb — real script, temp CONFIG_DIR -| Test | Result | -|------|--------| -| `prefix` (no arg) → `AI bridge prefix: ai` + usage hint | ✅ | -| `prefix bot` → writes `TELEGRAM_AI_PREFIX=bot` to telegram.env, rc 0 | ✅ | -| `prefix` after set → `AI bridge prefix: bot` | ✅ | -| `prefix 'a b'` and `prefix 'a.b'` → rc 1, validation error | ✅ | -| `prefix 1ok` → accepted (leading digit allowed) | ✅ | -| `--status` → `ai prefix: ` | ✅ | -| `prefix ai` → back to default | ✅ | - -### Dispatch + config UI -- `pos communication telegram listener prefix` and flat `pos communication telegram-listener prefix` both show the current prefix (repo `bin/pos` on PATH). ✅ -- `pos config telegram` (temp dir) renders `TELEGRAM_AI_PREFIX` with its description. ✅ - -### Gates -`bash -n` ×2 · `make gen` idempotent · `make check` OK · `make lint` 0 FAIL / 0 WARN. ✅ - -[DONE] - ---- - -## 4. Notes / Remaining Risks - -- **Matrix listener untouched** — it has its own hard-coded `ai …` bridge (session `matrix-`); the same treatment is a possible follow-up if wanted. -- The daemon still loads `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` once at start (unchanged); only the prefix hot-reloads. -- A prefix change applies to ALL chats the listener serves (there is one owner chat by design). -- Literal prefix matching means a prefix containing regex metacharacters would be matched literally (quoted), but validation restricts to word chars for predictability. - -[DONE] - ---- - -**Status: COMPLETE.** The `ai ` trigger is now `TELEGRAM_AI_PREFIX` (default `ai`), configurable via `pos communication telegram listener prefix ` or `pos config telegram`, applied live. \ No newline at end of file diff --git a/AgentsReport/builder/2026-08-29_telegram-prefix-map.md b/AgentsReport/builder/2026-08-29_telegram-prefix-map.md deleted file mode 100644 index 2fb3f84..0000000 --- a/AgentsReport/builder/2026-08-29_telegram-prefix-map.md +++ /dev/null @@ -1,43 +0,0 @@ -# Telegram listener: generic text-prefix → app map - -**TL;DR:** The listener now routes any non-command message ` ` to a user-configurable command with `` appended as ONE quoted argument — `opencode=opencode` turns "opencode check cpu" into `opencode "check cpu"`. Routing order per message: text-prefix map → built-in Gemini `ai` bridge → `/command` map → "Unknown command". `prefix` verb reworked to manage the map; `TELEGRAM_AI_PREFIX` is now set only via `pos config telegram`. - -## Steps - -- [DONE] Design — generalized the previous scalar AI-prefix into a full prefix→command map per the user's clarification ("each prefix call a app with pass 'text'"); kept the `/command` map and the Gemini bridge untouched, with the prefix map checked first so a mapped word can shadow `ai`. -- [DONE] Implement `bin/pos-communication-telegram-listener` (623→782): - - `PREFIX_FILE="$CONFIG_DIR/telegram_prefixes.env"` (chmod 600, hot-reloaded per message like the command map). - - `prefix_map_find(text)` — case-insensitive `^word[[:space:]](.*)$` (quoted-literal bash regex in scoped `nocasematch`); first file match wins; requires non-empty remainder; returns `command\x1fremainder`. - - `prefix_map_set/del/show` — same mktemp+mv pattern as the map functions; words validated `[A-Za-z0-9][A-Za-z0-9_-]*`; commands `bash -n`-checked via `check_syntax` (fixed latent `set -e` abort: `errs="$(...)" || err ...`). - - `run_and_reply(cmdline, msg_id, tmo, quiet)` — shared runner: `timeout bash -c`, empty→`OK`, non-zero→`exit N`+output, `@quiet`→no reply. `/command` map uses 60s (unchanged), prefix map 120s. - - `handle_message` — prefix bridge inserted after `/help|/start`, before the AI bridge; `/command` map block refactored onto `run_and_reply`. - - `prefix` verb: bare = list + bridge word; `prefix ` = map; `prefix ` = show one; `prefix -r ` = remove. Dispatch `prefix) prefix_cmd "${@:2}"`. - - `--status` lists prefix entries; usage() + header `# POS:` line updated. -- [DONE] Docs — POS.md listener rows + new Text-prefix map / AI bridge paragraphs; howto/communication.md text-prefix bullet; howto/ai.md trigger-word wording (now `pos config telegram`) + prefix-map/shadowing note; AGENT_Context regen via `make gen`. -- [DONE] AGENT_TODO.md Done entry appended (newest-last). -- [PENDING] Commit + push (next step after this report). - -## Verification - -- Routing harness `/tmp/prefix_map_routing_test.sh` — extracts the real listener functions (incl. new prefix map helpers + `run_and_reply`) with PATH stubs for `pos` and `opencode`; **27/27 PASS**: - - ai-bridge regression: `ai hello` → gemini ask, `ai /reset` → sessions reset, no-prefix → Unknown. - - `opencode check cpu` → app runs, remainder is ONE arg (`argc=1 arg1=`), output replied. - - Case-insensitive `OpEnCoDe`; bare `opencode` and `opencode ` (trailing space) fall through to Unknown. - - Mapped `ai` shadows the Gemini bridge and passes the remainder. - - No false match on partial prefixes (`o` vs `opencode`). - - `exit N` reply, empty→`OK`, `@quiet` suppression, `$TELEGRAM_CHAT_ID` expansion in templates. - - `/command` map regression through `run_and_reply` (`/health`, `/failx` exit 2, `@quiet`, `/help` list, unknown). -- CLI verb suite (real tool, temp `CONFIG_DIR`): set / show / remove / remove-missing (rc 0) / invalid word (rc 1) / invalid cmd `if then` (rc 1, nice `ERROR:` instead of raw `set -e` abort) / update-existing / `@quiet` template accepted / `--status` listing. -- Dispatch smoke via `bin/pos`: nested `pos communication telegram listener prefix …` and flat `pos communication telegram-listener prefix -r …` both work; bare `prefix` lists. -- `pos config telegram` (temp dir) still renders `TELEGRAM_AI_PREFIX` with description. -- Gates: `bash -n` clean; `make gen && make check` OK; `make lint` 0 FAIL / 0 WARN. - -## Notes / behavior choices - -- Bare `` (no trailing space) does NOT trigger a prefix app — falls through to the Gemini bridge / command map / Unknown, preserving the pre-existing `ai` edge behavior. -- Prefix apps run with the FULL remainder as one quoted argument (`printf '%q'`); a template wanting the text mid-line can place it anywhere since it is appended after the command. -- Because matching requires ``, two distinct map words can never both match one message — "first match wins" only matters for hand-edited files with unusual words; kept as documented file-order semantics. -- Text-prefix entries are NOT pushed to the bot command menu (`build_commands_json` reads only `MAP_FILE`) — they are text triggers, not `/`-commands. -- The old `prefix ` = set `TELEGRAM_AI_PREFIX` behavior is intentionally replaced; the word is set via `pos config telegram` (field already exists in the `telegram` scope) and displayed by bare `prefix` + `--status`. - -**Status: COMPLETE** (pre-commit). Next: commit + push. \ No newline at end of file diff --git a/AgentsReport/builder/2026-09-04_hf-downloader-implementation.md b/AgentsReport/builder/2026-09-04_hf-downloader-implementation.md deleted file mode 100644 index d377a2d..0000000 --- a/AgentsReport/builder/2026-09-04_hf-downloader-implementation.md +++ /dev/null @@ -1,102 +0,0 @@ -# Builder Report — `pos ai hf` (Hugging Face Model Downloader) - -**Date:** 2026-09-04 -**Status:** DONE - ---- - -## TL;DR - -Implemented `bin/pos-ai-hf` per the architecture report: a bash-only (curl + jq) Hugging Face model downloader with `search`/`download`/`list`/`remove` subcommands, `ai`-scope config (`HF_TOKEN`, `HF_DOWNLOAD_DIR`), auth headers, HTTP 429 retry, `curl -C -` resume, `.hf-meta` bookkeeping, and emoji output. Test harness (46 cases) green; all gates pass. - -| Item | Status | -|------|--------| -| `bin/pos-ai-hf` created | [DONE] | -| Syntax check | [DONE] | -| Test harness (46/46) | [DONE] | -| `make gen && make check` | [DONE] | -| `make lint` (0 FAIL, 0 WARN) | [DONE] | -| Doc updates (POS.md, AGENT_TODO.md) | [DONE] | - ---- - -## Step 1: Create `bin/pos-ai-hf` from template + implement full tool - -Implemented the full tool: config loader (env-var precedence over `ai.env`), `hf_api`/`hf_repo_files`/`hf_search` helpers, `cmd_search`/`cmd_download`/`cmd_list`/`cmd_remove`, usage/help, error handling, `.hf-meta` writing. Deps guards (`curl`/`jq`) before `--help`. `# POS: ai hf —`, `# POS_FLAGS`, `# POS_DEPS`, `# POS_CONFIG`, `# POS_EXAMPLES` headers present. Chmod 100755. - -[DONE] - -## Step 2: Syntax check (`bash -n bin/pos-ai-hf`) - -`bash -n bin/pos-ai-hf` → Syntax OK. - -[DONE] - -## Step 3: Test harness (`/tmp/opencode/hf-test/run-tests.sh`) - -Stub-PATH harness: stub `curl` routes by URL pattern to fixtures; 46 numbered tests covering argument parsing (missing/unknown/invalid), download (single file via `-o`, whole repo, `--gguf`, `--branch`, `--output`), search, list, remove, config/token handling, and output format (emoji, size header/table formats). - -Fixed two harness issues along the way: Test 22 size grep double-match (corrected regex), Test 40 token warning (empty config file + unset `HF_TOKEN` when the tool re-reads the config after `unset`). - -**Result:** 46/46 green, exit 0. - -[DONE] - -## Step 4: Gates — `make gen && make check && make lint` - -- `make gen` → `gen-docs: write OK` -- `make check` → `check-sync: OK` -- `make lint` → **0 FAIL, 0 WARN** - - Initially hit 1 WARN (`pos-ai-hf: file not referenced in DOC/POS.md`); added the `bin/pos-ai-hf` reference in the `### ai` `**File:**` line — lint green after. - -Verified generated output: docmap/tree/filetable updated in AGENT_Context, `completions/pos.bash` gained `ai-hf` flags and `ai` subcmd list, dispatch (`pos ai --help`, `pos help ai hf`, `pos ai hf --help`) all resolve. - -[DONE] - -## Step 5: Doc updates (POS.md, AGENT_TODO.md) - -- `DOC/POS.md`: added `bin/pos-ai-hf` to the `### ai` `**File:**` line, and a dedicated `pos ai hf` command table + auth/rate-limit/resume detail block under the ai section. -- `AGENT_TODO.md`: added a **Done** entry (newest-first, dated 2026-09-04) describing the tool, seam, config scope, verification. - -[DONE] - ---- - -## Verification Summary - -- `bash -n bin/pos-ai-hf` — OK -- `/tmp/opencode/hf-test/run-tests.sh` — 46/46 passed, exit 0 -- `make gen` — write OK -- `make check` — OK -- `make lint` — 0 FAIL, 0 WARN -- `git status --short` — only intended changes: `AGENT_TODO.md`, generated `DOC/AGENT_Context_Project.md`, `DOC/POS.md`, `completions/pos.bash`, new `bin/pos-ai-hf`, new report file. - -## Remaining Risks / Follow-up - -- No live-network smoke against the real `huggingface.co` was run (stub-based tests only); real end-to-end (auth 401/403, 429, 404, jq parse error paths) should be exercised manually on a machine with network access before first real use. -- `.hf-meta` uses a plain JSON object — no schema version field; acceptable for this scope. - -## Handoff - -Status: IMPLEMENTED - -Approved scope: -`bin/pos-ai-hf` (from `templates/pos-tool.sh`), `DOC/POS.md`, `AGENT_TODO.md`, test harness, report. No changes to `bin/pos-ai`, `bin/pos`, `preinstall.sh`, `lib/common.sh`, or INTERACTIVE_CMDS. - -Changes made: -New tool `bin/pos-ai-hf`; POS.md ai row + hf detail block; AGENT_TODO Done entry; generated docs/completions refreshed via `make gen`. - -Files changed: -`bin/pos-ai-hf` (new), `DOC/POS.md`, `AGENT_TODO.md`, `DOC/AGENT_Context_Project.md` (generated), `completions/pos.bash` (generated), `AgentsReport/builder/2026-09-04_hf-downloader-implementation.md` (new). - -Verification performed: -`bash -n`, 46/46 stub tests, `make gen && make check && make lint` (0 FAIL, 0 WARN). - -Scope compliance: -In-scope changes only; no out-of-scope changes. - -Recommended next agent: -Reviewer - -Reason: -Implementation complete and gates green; needs independent adversarial review before acceptance. diff --git a/AgentsReport/builder/2026-09-04_llamacpp-server-implementation.md b/AgentsReport/builder/2026-09-04_llamacpp-server-implementation.md deleted file mode 100644 index b38b26c..0000000 --- a/AgentsReport/builder/2026-09-04_llamacpp-server-implementation.md +++ /dev/null @@ -1,37 +0,0 @@ -# Builder Report: `pos ai server` — llama.cpp Inference Server - -## TL;DR - -- **Status:** IMPLEMENTED -- **Files created:** `bin/pos-ai-server`, `lib/ai-providers/llamacpp.sh` -- **Files modified:** `bin/pos-ai` (4 case additions + POS_CONFIG header), `config/ai.env` (LLAMACPP_* docs), `DOC/POS.md` (ai server docs) -- **Test harness:** `/tmp/opencode/llamacpp-test/run-tests.sh` — 87/87 passing -- **Verification:** `make gen && make check && make lint` = 0 FAIL, 0 WARN - -## Step 1: Create `bin/pos-ai-server` -[DONE] — chmod 100755, syntax check passed, all conventions followed (set -euo pipefail, deps guards before --help, POS/POS_SUBCMDS/POS_FLAGS/POS_DEPS headers, common.sh source chain, err/log/warn/ok helpers, config seam guards) - -## Step 2: Create `lib/ai-providers/llamacpp.sh` -[DONE] — PROVIDER_CONFIG header present, 4-function contract (provider_name, provider_default_model, provider_generate, provider_models_list), syntax check passed - -## Step 3: Modify `bin/pos-ai` — 4 case additions -[DONE] — resolve_key: llamacpp returns 0 (no key needed), resolve_model: reads LLAMACPP_MODEL, cmd_providers: always "configured", require_key: llamacpp case (no key needed). POS_CONFIG header updated with LLAMACPP_* keys. Syntax check passed. - -## Step 4: Modify `config/ai.env` — add LLAMACPP_* docs -[DONE] — All 6 LLAMACPP_* config keys documented with defaults - -## Step 5: Add POS.md documentation -[DONE] — ai section updated with file references, server command table, flags, and config keys - -## Step 6: Create test harness -[DONE] — `/tmp/opencode/llamacpp-test/run-tests.sh` with 45 test groups (87 individual assertions), 0 failures. Tests cover: help/dispatch, error handling, start (config/arg/fallback), systemd unit generation (all fields), GPU detection (CPU/CUDA), model resolution (config/arg/relative/TTYPick), stop/status/models/logs, dry-run mode, flag overrides, POS headers, deps guards, pos-ai modifications. - -## Step 7: Verification -[DONE] — All gates pass: -- `bash -n bin/pos-ai-server` — OK -- `bash -n lib/ai-providers/llamacpp.sh` — OK -- `bash -n bin/pos-ai` — OK -- Test suite: 87/87 passed -- `make gen` — OK -- `make check` — OK -- `make lint` — 0 FAIL, 0 WARN diff --git a/AgentsReport/builder/2026-09-04_media-grab-implementation.md b/AgentsReport/builder/2026-09-04_media-grab-implementation.md deleted file mode 100644 index 06e4936..0000000 --- a/AgentsReport/builder/2026-09-04_media-grab-implementation.md +++ /dev/null @@ -1,185 +0,0 @@ -# Builder Report — `pos media grab` Implementation - -**Date:** 2026-09-04 -**Status:** IMPLEMENTED - ---- - -## TL;DR - -- **Status:** IMPLEMENTED -- **Files created:** `bin/pos-media-grab` (chmod 100755) -- **Files modified:** `bin/pos-communication-telegram-listener`, `DOC/POS.md`, `AGENT_TODO.md` -- **Test suite:** `/tmp/opencode/media-grab-test/run-tests.sh` — **28 test cases / 70 assertions, 0 failed** -- **Gates:** `bash -n` on both files ✅; `make gen && make check` green ✅; `make lint` 0 FAIL / 0 WARN ✅ -- **Report path:** `./AgentsReport/builder/2026-09-04_media-grab-implementation.md` - ---- - -## Step 1: Create `bin/pos-media-grab` - -Created a new pos tool at `bin/pos-media-grab` (chmod 100755, `set -euo pipefail`). - -**Headers (right after shebang):** -- `# POS: media grab — Auto-download URL as audio or video (classify + route)` -- `# POS_FLAGS: --audio --video --best --worst --output --no-playlist --cookies --dry-run` -- `# POS_CONFIG: grab | grab.env | GRAB_DEFAULT=:Default mode for unknown domains (video or audio, default video)` - -**Implemented functions/behavior:** -- `classify_url()` — domain-based regex classification (full Decision 2 table) -- `usage()` — full help with examples -- Main arg parsing loop (`--audio`, `--video`, `--best`, `--worst`, `--output`, `--no-playlist`, `--cookies`, `--dry-run`, `-h|--help`) -- Single positional URL; non-HTTP → `err "not a valid URL"` -- Routes to `pos media mp3` (audio) or `pos media mp4` (video) -- mp4 route adds `--best` by default, `--worst` if user passes `--worst` -- All flag values forwarded to mp3/mp4 as-is -- After download: `yt-dlp --print title --print duration_string` for metadata, `stat --printf='%s'` for file size, formatted summary -- `--dry-run` prints the command, doesn't execute -- Captures stderr from delegated command, prints `❌ Download failed: ` -- Config: `load_grab_config()` reads `GRAB_DEFAULT` from `~/.config/linux_post_install/grab.env` -- No deps guards (delegates to mp3/mp4 which handle their own) -- Sources `lib/common.sh` via the standard fallback chain - -[`bin/pos-media-grab`](file:///home/unknown/projects/Linux_post_install/bin/pos-media-grab) -[DONE] - -## Step 2: Modify `bin/pos-communication-telegram-listener` - -**Change 1 — `url_detect` function** (added before `handle_message`, after `strip_markdown`): -- Extracts first http(s) URL, strips trailing punctuation (`.,\)!?:;` and `>*`) -- Returns 0 + prints URL on success, 1 if no URL found - -**Change 2 — URL routing step** in `handle_message` (after prefix map block, before AI bridge check): -```bash -# URL detect: bare HTTP(S) URLs → pos media grab -local grab_url -if grab_url="$(url_detect "$text")"; then - log "grab: $grab_url" - run_and_reply "pos media grab --best \"$grab_url\"" "$msg_id" 600 - return -fi -``` - -Routing chain is now: `/help|/start` → prefix map → **URL detect** → AI bridge → command map → Unknown. - -[DONE] - -## Step 3: Create test harness + run tests - -Created `/tmp/opencode/media-grab-test/` with: -- `run-tests.sh` — main test runner (28 test cases / 70 assertions) -- `stubs/pos` — fake dispatcher -- `stubs/pos-media-mp3`, `stubs/pos-media-mp4` — fake download tools that log args + create dummy files (mp3 validates `--cookies` like the real tool) -- `stubs/yt-dlp` — fake yt-dlp that prints metadata and creates dummy downloads -- `stubs/stat` — fake stat returning fixed sizes - -**Test results: 28/28 cases, 70/70 assertions, 0 failed.** - -| # | Test | Result | -|---|------|--------| -| 1 | YouTube Music URL → mp3 | ✅ | -| 2 | YouTube video URL → mp4 --best | ✅ | -| 3 | YouTube youtu.be short → mp4 --best | ✅ | -| 4 | YouTube shorts → mp4 --best | ✅ | -| 5 | SoundCloud → mp3 | ✅ | -| 6 | Bandcamp → mp3 | ✅ | -| 7 | Vimeo → mp4 --best | ✅ | -| 8 | Unknown domain → mp4 --best (default) | ✅ | -| 9 | `--audio` override → mp3 | ✅ | -| 10 | `--video` override → mp4 | ✅ | -| 11 | `--worst` → mp4 --worst | ✅ | -| 12 | `--best` explicit → mp4 --best | ✅ | -| 13 | `--dry-run` prints command, no download | ✅ | -| 14 | No URL → usage (exit 0) | ✅ | -| 15 | `--help` → usage | ✅ | -| 16 | Invalid URL → error | ✅ | -| 17 | `GRAB_DEFAULT=audio` (config file) → mp3 | ✅ | -| 18 | `--output` override → passed to mp3 | ✅ | -| 19 | HTTP (not HTTPS) URL → mp4 | ✅ | -| 20 | `--no-playlist` → passed to mp4 | ✅ | -| 21 | `--cookies` missing file → error | ✅ | -| 22 | Unknown option → error | ✅ | -| 23 | FTP URL rejected → error | ✅ | -| 24 | `--audio` + `--video` → error | ✅ | -| 25 | `--best` + `--worst` → error | ✅ | -| 26 | Output contains 🎬 + metadata | ✅ | -| 27 | Output contains 📁 file info | ✅ | -| 28 | Twitch URL → mp4 --best | ✅ | - -[DONE] - -## Step 4: Gates — syntax, gen, check, lint - -- `bash -n bin/pos-media-grab` ✅ -- `bash -n bin/pos-communication-telegram-listener` ✅ -- `make gen && make check` — green ✅ -- `make lint` — **0 FAIL, 0 WARN** ✅ - -`make gen` regenerated `DOC/AGENT_Context_Project.md` (tree, dispatch, filetable: added `pos-media-grab`, bumped listener 782→805 and completions 308→309) and `completions/pos.bash` (added `media-grab` flags + `grab` config scope). - -**Note on `DOC/POS.md`:** the lint gate requires every `bin/pos-*` tool to be referenced in `DOC/POS.md`. The Architect's report listed POS.md as a doc to update, so I added a `pos media grab` row to the media table (hand-maintained, lint-required). - -[DONE] - -## Step 5: Update `AGENT_TODO.md` - -Added the completed `pos media grab` entry to the **Done** section (newest-first, dated 2026-09-04). No matching "Next" item existed (the only Next item — wire alerting into more tools — is unrelated to this task). - -[DONE] - ---- - -## Deviations from Architect design - -| Area | Architect spec | Builder change | Justification | -|------|---------------|----------------|---------------| -| POS.md | Listed as "Files to Update (Docs)" | Added `pos media grab` row to media table | Required by `make lint` gate (lint WARNS on tools not referenced in POS.md) | -| No-URL usage | brief says "No positional → usage (exit 1)" | usage() exits 0 (project convention) | All existing tools' `usage()` calls `exit 0` — the exit-1 spec conflicts with the established project pattern | - -## Remaining uncertainty - -- The `find`-based file-location logic (for summary) picks the most recent matching file in the output dir — in a rare scenario with many recent files it could select a stale match, but for the primary single-download use case this is reliable. -- `--cookies` existence validation is delegated to mp3/mp4 (which validate it) — grab forwards as-is per spec. Test 21 verifies the failure path through the delegated tool. - -## Recommended next agent - -Reviewer - -Reason: Implementation is complete and needs independent adversarial review before acceptance. - ---- - -Status: IMPLEMENTED - -**Approved scope:** `bin/pos-media-grab` (new tool), listener modification (add `url_detect` + URL routing step), doc updates (POS.md, AGENT_Context via `make gen`, AGENT_TODO.md). No changes to mp3/mp4. - -**Changes made:** -1. New `bin/pos-media-grab` — domain-based URL classifier that delegates to `pos media mp3`/`pos media mp4`, adds `--best` for non-interactive video, prints clean output. Config scope `grab`. -2. `bin/pos-communication-telegram-listener` — added `url_detect()` + URL routing step (bare http(s) → `pos media grab --best`, 600s timeout). -3. `DOC/POS.md` — added media table row (lint-required). -4. `AGENT_TODO.md` — moved completed work to Done. - -**Files changed:** -- `bin/pos-media-grab` (new, chmod 100755) -- `bin/pos-communication-telegram-listener` (modified) -- `DOC/POS.md` (modified) -- `DOC/AGENT_Context_Project.md` (regenerated via make gen) -- `completions/pos.bash` (regenerated via make gen) -- `AGENT_TODO.md` (modified) - -**Verification performed:** -- `bash -n` on both files — OK -- Test suite `/tmp/opencode/media-grab-test/run-tests.sh` — 28 cases / 70 assertions / 0 failed -- `make gen && make check` — green -- `make lint` — 0 FAIL, 0 WARN - -**Project validation:** All required gates pass. - -**Scope compliance:** In-scope changes only. No changes to mp3/mp4, no feature flags, no scope expansion. - -**Remaining risks:** -- The `find`-based file-location for the summary picks the most recent matching file in the output dir; in rare multi-download scenarios it could select a stale match (reliable for the primary single-download use case). -- `--cookies` file existence is validated by the delegated mp3/mp4 (not grab) — per spec, grab forwards as-is. -- The installed `/usr/local/bin/pos-config` won't know the `grab` scope until `install.sh` refresh — this is a normal installed-vs-repo staleness, not a code bug. - -**Changes made by Builder:** In-scope implementation only. diff --git a/AgentsReport/builder/2026-09-04_pos-ai-hf-gguf-jq-bug-fix.md b/AgentsReport/builder/2026-09-04_pos-ai-hf-gguf-jq-bug-fix.md deleted file mode 100644 index a20dacc..0000000 --- a/AgentsReport/builder/2026-09-04_pos-ai-hf-gguf-jq-bug-fix.md +++ /dev/null @@ -1,120 +0,0 @@ -# Builder Report — `pos-ai-hf` GGUF/jq bug fix (2026-09-04) - -## TL;DR - -- **Status:** IMPLEMENTED -- **Scope:** fix `jq: endswith() requires string inputs` crash in `pos ai hf download … --gguf` per Detective spec (AgentsReport/detective/2026-09-04_pos-ai-hf-gguf-jq-bug.md, Step 4); Changes 1-3 implemented exactly; harness at `/tmp/opencode/hf-test2/`; no commit/push. -- **Files changed (repo):** `bin/pos-ai-hf` (+17/-4); `DOC/AGENT_Context_Project.md` (2-line auto-gen filetable row, only the `pos-ai-hf` line count 495→506 — `make gen` output; no GEN block content changed). -- **Verification:** `bash -n` OK; harness 9/9 PASS; `make gen/check/lint` green (`0 FAIL, 0 WARN`); live API: normalize→13 records (0 nulls, 0 bad sizes), --gguf→exactly 10, README/LICENSE/.gitattributes excluded; real tool path: single-file `LICENSE` download OK (7.2 KB, `.hf-meta` correct); mode-aware empty messages verified live (exit 1 unchanged). -- **NOT committed.** Working-tree changes: `bin/pos-ai-hf`, `DOC/AGENT_Context_Project.md` (+1/-1 line-count row), untracked Detective report (pre-existing). - -## Step 1: Read Detective report + confirm scope — [DONE] - -Read full report (fix spec Step 4, edge cases Step 5, harness spec Step 6, verification Step 7). Confirmed fixtures exist: `/tmp/opencode/qwen-tree.json` (13 files, no rfilename), `/tmp/opencode/sd-tree.json` (8 files + 7 dirs). - -## Step 2: Implement Change 1 — normalize tree response in `hf_repo_files()` — [DONE] - -`bin/pos-ai-hf:201-207` — primary `/tree` path now pipes through the normalize jq instead of echoing raw: - -```bash -# Tree API returns {type,path,size,oid[,lfs]} per entry — normalize to the -# {rfilename,size} shape the rest of the pipeline expects (same as fallback). -# Skip "directory" entries: they have no resolvable file URL. -printf '%s' "$result" | jq '[.[] | select(.type == "file") | {rfilename: .path, size: (.size // 0)}]' -``` - -Fallback sibling path (line ~213) untouched — already emits `{rfilename, size}`. - -## Step 3: Implement Change 2 — defense-in-depth guard in `--gguf` filter — [DONE] - -`bin/pos-ai-hf:339`: - -```bash -filtered_files="$(printf '%s' "$files_json" | jq -c '[.[] | select((.rfilename | type) == "string" and (.rfilename | endswith(".gguf")))]')" -``` - -## Step 4: Implement Change 3 — mode-aware empty results — [DONE] - -`bin/pos-ai-hf:347-356` — replaced `[ "$file_count" -gt 0 ] || err "No files to download"` with `if [ "$file_count" -eq 0 ]` branch: -- single-file mode: `err "File not found: $filename in $repo_id (branch: ${branch})"` -- `--gguf` mode: `err "No .gguf files found in $repo_id${branch:+ (branch: $branch)} — try without --gguf"` -- generic: `err "No files to download"` (unchanged text) - -Exit semantics unchanged (same `err` path, exit 1). - -## Step 5: Build harness `/tmp/opencode/hf-test2/` — [DONE] - -- `fixtures/tree-files.json` = copy of `/tmp/opencode/qwen-tree.json` (13 files, rfilename ABSENT, 10 .gguf) -- `fixtures/tree-with-dirs.json` = copy of `/tmp/opencode/sd-tree.json` (8 files + 7 `type:"directory"`) -- `fixtures/tree-empty.json` = `[]` -- `fixtures/tree-nogguf.json` = `[{"type":"file","path":"README.md","size":100}]` -- `run-tests.sh`: 9 assertions per report Step 6 (t_tree_normalize, t_gguf_on_normalized, t_single_file, t_all_files_passthrough, t_empty, t_nogguf, t_defense_guard, t_dirs_excluded, t_code_sync). Deliberately does NOT `source` bin/pos-ai-hf (top-level dispatch executes; no-args → usage → exit 0). NORM/GGUF_FILTER/FN_FILTER duplicated verbatim; `grep -F` drift-guards catch divergence from the file. - -## Step 6: Verification budget — [DONE] - -1. `bash -n bin/pos-ai-hf` → OK -2. `bash /tmp/opencode/hf-test2/run-tests.sh` → **9 passed, 0 failed** (output captured in Step 5 run) -3. `make gen && make check && make lint` → gen OK, check-sync OK, lint **0 FAIL, 0 WARN**; regenerated: `DOC/AGENT_Context_Project.md` line-count row only (495→506); no GEN:START/END block changes -4. LIVE validation (no download): fetched `https://huggingface.co/api/models/Qwen/Qwen2.5-3B-Instruct-GGUF/tree/main`; normalize jq → `{count:13, nulls:0, badsizes:0}`; --gguf filter → length 10; paths = 10 `qwen2.5-3b-instruct-*.gguf` (fp16×2 + q2_k..q8_0); README/LICENSE/.gitattributes excluded; pre-fix crash `endswith() requires string inputs` reproduced for contrast on raw tree -5. Real tool-path proof (no multi-GB download): - - `pos ai hf download Qwen/Qwen2.5-3B-Instruct-GGUF LICENSE --output /tmp/hf-small` → exit 0, `📥 Downloaded: Qwen/Qwen2.5-3B-Instruct-GGUF/LICENSE (7.2 KB)`, `.hf-meta` `"files": ["LICENSE"]`, real 7388-byte file present. NOTE: first attempt via `bin/pos` hit the stale **installed** `/usr/local/bin/pos-ai-hf` (PATH precedence) which silently no-matched → reproduced the pre-fix bug; installed copy is now byte-identical to repo and the same command succeeds. Environment detail, not a code issue. - - `--gguf` no-gguf repo: `… distilbert/distilbert-base-uncased --gguf` → `ERROR: No .gguf files found in distilbert/distilbert-base-uncased (branch: main) — try without --gguf`, exit 1 (no crash, no download) - - single-file not-found: `… totally-missing-file` → `ERROR: File not found: totally-missing-file in Qwen/Qwen2.5-3B-Instruct-GGUF (branch: main)`, exit 1 - -## Step 7: Final diff review + handoff — [DONE] - -`git diff --stat`: `bin/pos-ai-hf | 17 ++++-----` (only the 3 sanctioned hunks), `DOC/AGENT_Context_Project.md | 2 +-` (gen line-count row). `git status`: no staged/committed changes; NOT committed or pushed. - -## Handoff - -Status: **IMPLEMENTED** - -Approved scope: the 3 changes in Detective fix spec Step 4; harness at `/tmp/opencode/hf-test2/`; verification per Step 7. Only `bin/pos-ai-hf` changed in repo (+ `make gen` line-count row in DOC/AGENT_Context_Project.md). - -Remaining risks / follow-ups (out of scope, flagged by Detective): -- `hf_api` lacks `curl -L` → 307-redirect alias repos (e.g. runwayml) still fail (pre-existing). -- Tree endpoint non-recursive → subdirectory files not listed (pre-existing semantics). -- `/usr/local/bin` installed copy was stale at run time (auto-synced later); real deployers should reinstall. - -Recommended next agent: **Tester** — the /tmp/opencode/hf-test2 harness is fixture-based and ready for adoption into the repo test suite if the project chooses (decision: Architect); otherwise Reviewer for acceptance of the 3-hunk fix. - -Changes made by Builder: as listed above; nothing else touched. - -## Harden+verification — error-object hardening (2026-09-04, Orchestrator follow-up) — [DONE] - -Previous implementation APPROVED. Orchestrator/Reviewer found an additional crash class: `printf '%s' '{"error":"x"}' | jq '[.[] | select(.type == "file") | …]'` → `jq: error: Cannot index string with string "type"` rc 5 — `.[]` on an object iterates its VALUES; the string `"x"` then gets indexed with `.type`. Verified present on BOTH normalize paths pre-change (primary: `Cannot index string…`; fallback: `Cannot iterate over null (null)` on `.siblings`). - -### Change 4 — object-safe normalize (primary), object-safe fallback (new) - -`bin/pos-ai-hf:205` (primary, now object-guarded; jq `and` short-circuits so `.type` is never evaluated on non-objects): - -```bash -printf '%s' "$result" | jq '[.[] | select(type == "object" and .type == "file") | {rfilename: .path, size: (.size // 0)}]' -``` - -`bin/pos-ai-hf:213` (fallback, previously unguarded — same crash class; now `[]?` suppresses null iteration + `select(type == "object")` skips junk elements + `(rfilename // "")` keeps the shape contract string-safe for nulls): - -```bash -printf '%s' "$fallback" | jq '[.siblings[]? | select(type == "object") | {rfilename: (.rfilename // ""), size: (.size // 0)}]' -``` - -- On `{"error":"x"}`: primary → `[]` rc 0; fallback → `[]` rc 0 (both were rc 5 before). -- On real fixtures: 13-file tree → 13 records, 0 nulls (identical to pre-hardening); dirs fixture → 8 (dirs dropped); real metadata qwen-meta.json → 13 records, string rfilename, numeric size. -- Pathological `{"rfilename":42}` in siblings passes `// ""` unchanged (42 is truthy → kept) — non-null non-string rfilename still possible in the fallback shape; the `--gguf` filter's `type == "string"` guard prevents the crash class there, and single-file select simply won't match. Flagged as accepted residual risk (suggested-form semantics per Orchestrator). -- Line count unchanged (506) → `make gen` produced no further DOC change beyond the already-tracked 495→506 line-count row. - -### Harness update - -`/tmp/opencode/hf-test2/run-tests.sh`: -- `NORM` updated to hardened primary form; new `FB_NORM` duplicated verbatim. -- New fixture `fixtures/meta-siblings.json` = copy of `/tmp/opencode/qwen-meta.json` (13 siblings, real metadata shape). -- New assertions: `t_error_object_normalize` (`{"error":"x"}` → `[]` rc 0), `t_fallback_normalize` (real metadata → 13 records, string rfilename, numeric size), `t_error_object_fallback` (`{"error":"x"}` → `[]` rc 0). -- `t_code_sync` drift-guards updated: greps `select(type == "object" and .type == "file")` (primary) and `select(type == "object")` (fallback) in addition to the GGUF guard + fn filter. - -### Harden verification results - -1. `bash -n bin/pos-ai-hf` → OK -2. `bash /tmp/opencode/hf-test2/run-tests.sh` → **12 passed, 0 failed** (was 9; +3 new assertions) -3. `make gen && make check && make lint` → gen OK, check-sync OK, **0 FAIL, 0 WARN**; `git diff --stat`: `bin/pos-ai-hf | 19 ++++---` (4 sanctioned hunks: normalize + fallback + gguf guard + message branch + comment), `DOC/AGENT_Context_Project.md | 2 +-` (line-count row from prior gen; unchanged by this pass) -4. LIVE (real API, no download): Qwen tree → hardened normalize `{"count":13,"nulls":0}`; hardened `--gguf` filter → 10; README/LICENSE/.gitattributes excluded → `OK` -5. Still NOT committed; only intended files modified (bin/pos-ai-hf, DOC line-count row) + untracked reports. \ No newline at end of file diff --git a/AgentsReport/builder/2026-09-04_ytsync-channel-handle-fix.md b/AgentsReport/builder/2026-09-04_ytsync-channel-handle-fix.md deleted file mode 100644 index 60dcec0..0000000 --- a/AgentsReport/builder/2026-09-04_ytsync-channel-handle-fix.md +++ /dev/null @@ -1,78 +0,0 @@ -# Builder Report: ytsync channel-handle fix - -Date: 2026-09-04 -Agent: Builder -Status: IMPLEMENTED - -## TL;DR - -- **Scope:** Single-file bug fix in `bin/pos-media-ytsync` — three changes (new helper + probe canonicalization + entry filter) -- **Files changed:** `bin/pos-media-ytsync` (+ `DOC/AGENT_Context_Project.md` line-count bump from `make gen`) -- **Baseline:** 12 PASS / 5 FAIL on unfixed script -- **Result:** 32 PASS / 0 FAIL; `make gen`/`check`/`lint` all green; live dry-run shows 151 real videos; NOT committed -- **Status: IMPLEMENTED** - -## Step 1: Implement `canonical_channel_url()` helper - -Insert after `classify_url()` (after line 176), before `sanitize_component()`. - -[DONE] - -## Step 2: Wire canonicalization into `run_probe()` - -Add `url="$(canonical_channel_url "$url")"` after `local url="$1"` in `run_probe()`. - -[DONE] - -## Step 3: Add entry filter in `collect_entries()` - -(a) Filter `.entries[]` to watchable URLs only -(b) Guard single-object fallback with `_type == "video"` check - -[DONE] - -## Step 4: Syntax check - -`bash -n bin/pos-media-ytsync` - -[DONE] - -## Step 5: Test harness — all assertions pass - -Baseline was 12 PASS / 5 FAIL. After fix: **32 PASS / 0 FAIL** (Detective's 17 logical checks; harness counts 32 check calls — all green). - -[DONE] - -## Step 6: Gates — make gen && make check && make lint - -`make gen` OK (regenerated `DOC/AGENT_Context_Project.md` — bumped hand-maintained line-count row for `bin/pos-media-ytsync` 1191 → 1213). `make check` OK. `make lint` → `0 FAIL, 0 WARN`. - -[DONE] - -## Step 7: Live dry-run — 3Blue1Brown sync - -Seeded `/tmp/opencode/ytsync-live/` with a copy of the real registry (bare-handle URL). Ran: -`YTSYNC_STATE_DIR=/tmp/opencode/ytsync-live YTSYNC_VIDEOS_DIR=/tmp/opencode/ytsync-live/vids bin/pos-media-ytsync sync --dry-run` - -Output (exact match to expected): -``` -Source : https://www.youtube.com/@3blue1brown ← stored URL unchanged (no migration) -Resolved : 3Blue1Brown (channel · 151 videos) -New : 151 would be downloaded (0 already present) - But what is cross-entropy? | Compression is Intelligence Part 2.mp4 - Reinventing Entropy | Compression is Intelligence Part 1.mp4 - ... - … 146 more -``` -Real video titles, not Videos/Live/Shorts tabs. Wrote nothing (archive empty, vids empty, registry unchanged). - -[DONE] - -## Step 8: One real download proof - -Downloaded exactly one entry (`GlYgs6v2YfU` = "But what is cross-entropy?") standalone via -`yt-dlp -o /tmp/opencode/ytsync-live/test.%(ext)s https://www.youtube.com/watch?v=GlYgs6v2YfU --no-playlist`. - -yt-dlp resolved the video (not "[youtube] : This video is unavailable"), pulled metadata, and began streaming (~154 MB / 31% of a 471 MiB file before the 180s tool timeout). Partial file cleaned up. This proves the dry-run probe's entry ids are valid and fetchable. - -[DONE] diff --git a/AgentsReport/builder/2026-09-05_enhanced_pos_ai_tools.md b/AgentsReport/builder/2026-09-05_enhanced_pos_ai_tools.md deleted file mode 100644 index 6d86e16..0000000 --- a/AgentsReport/builder/2026-09-05_enhanced_pos_ai_tools.md +++ /dev/null @@ -1,91 +0,0 @@ -# Enhanced POS AI Tools Implementation Report - -## TL;DR - -Successfully implemented enhanced POS AI tools with the following key features: - -1. **Enhanced `pos ai hf`**: - - Added `info` and `files` commands - - Enhanced download capabilities with include/exclude patterns - - Added revision support (`--revision`) - - Improved progress reporting - - Added cache management framework - -2. **Enhanced `pos ai server`**: - - Added detailed GPU configuration options (`--gpu-layers`, `--gpu-threads`, `--tensor-split`) - - Added memory context controls (`--mmap`, `--mlock`, `--kv-cache`) - - Added performance tuning options (`--batch-size`, `--ubatch-size`) - - Added sampling parameters (`--temperature`, `--top-k`, `--top-p`, `--repetition-penalty`) - - Added server configuration options (`--metrics`, `--health`, `--slots`) - - Implemented version awareness - -All changes maintain backward compatibility and follow existing code conventions. - -## Step 1: Enhanced `pos ai hf` Implementation - -### Added New Commands -- **Info Command**: `pos ai hf info ` - Shows repository metadata including downloads, likes, tags, description, author, and creation dates -- **Files Command**: `pos ai hf files ` - Lists all repository files with sizes and metadata -- **Cache Command**: Basic framework for cache management - -### Enhanced Download Capabilities -- **Include/Exclude Patterns**: Added `--include` and `--exclude` flags with glob support for filtering files during download -- **Revision Support**: Added `--revision` flag for targeting specific commits, tags, or branches -- **Improved Progress Reporting**: Enhanced download progress with better feedback and error handling -- **Pattern Filtering**: Supports filtering by file patterns during download - -## Step 2: Enhanced `pos ai server` Implementation - -### GPU Configuration -- **Detailed GPU Support**: Added `--gpu-layers`, `--gpu-threads`, and `--tensor-split` for advanced GPU offloading -- **Device Selection**: Improved GPU detection and automatic configuration - -### Memory and Context Management -- **Memory Allocation**: Added `--mmap` and `--mlock` for memory mapping and locking -- **KV Cache Configuration**: Added `--kv-cache` for custom KV cache sizing -- **Context Size Control**: Enhanced `--ctx-size` control with better validation - -### Performance Tuning -- **Batch Size Configuration**: Added `--batch-size` and `--ubatch-size` for processing configuration -- **Continuous Batching**: Support for batch processing options - -### Sampling Controls -- **Advanced Sampling**: Added `--temperature`, `--top-k`, `--top-p`, and `--repetition-penalty` for improved sampling behavior - -### Server Configuration -- **Endpoint Configuration**: Added `--metrics`, `--health`, and `--slots` for enhanced server configuration -- **Version Awareness**: Added `detect_llama_version()` and `validate_server_features()` functions for version detection and feature validation - -## Implementation Details - -### Files Modified -1. `bin/pos-ai-hf` - Enhanced with new commands and download capabilities -2. `bin/pos-ai-server` - Enhanced with new GPU, memory, and performance options - -### Backward Compatibility -- All existing commands and flags continue to work exactly as before -- New flags are optional and don't affect existing workflows -- Default behavior unchanged -- Configuration files remain compatible - -### Code Quality -- Follows existing project conventions and patterns -- Consistent error handling and messaging -- Proper usage documentation with examples -- Modular code structure with clear separation of concerns -- Comprehensive help text with examples - -## Verification - -The implementation has been tested to ensure: -- All existing functionality remains intact -- New commands properly parse arguments and display usage information -- Help text displays correctly with updated examples -- Error messages are descriptive and helpful -- Scripts are executable with proper shebangs - -All checks and tests pass: -- `make check` - OK -- `make lint` - 0 FAIL, 0 WARN - -[COMPLETE] \ No newline at end of file diff --git a/AgentsReport/builder/2026-09-06_ai-cost-window.md b/AgentsReport/builder/2026-09-06_ai-cost-window.md deleted file mode 100644 index 66074d7..0000000 --- a/AgentsReport/builder/2026-09-06_ai-cost-window.md +++ /dev/null @@ -1,226 +0,0 @@ -# Builder Report — AI cost & session-window design - -**Date:** 2026-09-06 -**HEAD:** 8ce5479 (clean tree; HEAD unchanged — implementation not committed) -**Design source:** `./AgentsReport/architect/2026-09-06_ai-cost-window-design.md` - -## TL;DR - -- **Status: IMPLEMENTED** (worktree changes uncommitted) -- **Files changed:** `bin/pos-ai`, `lib/ai-providers/openrouter.sh`, `lib/ai-providers/gemini.sh`, `DOC/POS.md`, `DOC/HOWTO.md`, `DOC/AGENT_Context_Project.md` (6 files, +18/−11; `make gen` auto-updated the `bin/pos-ai` filetable row 705→709) -- **Smoke:** provider-body cap (openrouter + gemini, default 2048 + custom 512) all pass; session_push lazy resolution + numeric guard all pass (incl. boundary 40, `10`, non-numeric/0/negative/empty fallback) -- **Gates:** `make gen` idempotent, `make check` OK, `make lint` 0 FAIL 0 WARN, `make test` 17/17 files + 299/299 checks, `git diff --check` clean, `bash -n` all touched scripts OK -- **Out-of-scope:** no changes to llamacpp.sh, pos-ai-server, aliases, other tools, tests, AGENT_TODO.md - ---- - -## Step 1: POS_CONFIG header — add AI_MAX_TOKENS and AI_SESSION_TURNS - -Append two entries to the `@General` section of `# POS_CONFIG:` header at `bin/pos-ai:6`. - -- `AI_MAX_TOKENS=num:Max output tokens per request (default 2048; OpenRouter/Gemini cost cap)` -- `AI_SESSION_TURNS=num:Session message cap — 2 per exchange (default 40 = 20 exchanges; 10 = last 5)` - -Wording matches the design report §Decision 3 exactly. `num:` type reuses the existing `LLAMACPP_CTX_SIZE=num:…` pattern in the same header. - -[DONE] - -## Step 2: session_push lazy resolution - -Change `session_push()` (`bin/pos-ai:285-292`) to resolve `AI_SESSION_TURNS` lazily with numeric guard. - -```bash -local n="${AI_SESSION_TURNS:-$MAX_SESSION_TURNS}" -[[ "$n" =~ ^[0-9]+$ ]] && (( n >= 1 )) || n="$MAX_SESSION_TURNS" -``` - -- `^[0-9]+$` regex + `>= 1` → fallback on: empty, non-numeric, `0`, negative. -- Line 25 `MAX_SESSION_TURNS=40` untouched (help text `:67` still accurate). -- jq pruning now uses `--argjson n "$n" ... .[-$n:]` (no string interpolation into the filter). - -[DONE] - -## Step 3: Add Config help lines for the two new vars - -Added to the `Config:` block in `pos ai --help` (`bin/pos-ai:95-96`): -`AI_MAX_TOKENS` + `AI_SESSION_TURNS` lines, matching the design's recommended `Config:` help documentation. - -[DONE] - -## Step 4: openrouter.sh — add max_tokens to body - -`lib/ai-providers/openrouter.sh:22-23`: body now includes `max_tokens:($mt|tonumber)` via `--arg mt "${AI_MAX_TOKENS:-2048}"`. Empty/unset env → jq receives the literal `2048` (the `:-2048` default is a numeric literal, so there is no empty-string `tonumber` failure path). - -[DONE] - -## Step 5: gemini.sh — add generationConfig.maxOutputTokens - -`lib/ai-providers/gemini.sh:17-19`: body now includes `generationConfig: {maxOutputTokens: ($mt|tonumber)}` via the same `--arg mt "${AI_MAX_TOKENS:-2048}"`. `systemInstruction` merging (`:20-23`) unchanged and composes with the new top-level field. - -[DONE] - -## Step 6: Doc updates - -- `DOC/POS.md:93-95` ai.env table: two new rows after `AI_SYSTEM_PROMPT` (`AI_MAX_TOKENS` no/`2048`/"Max output tokens per request (OpenRouter/Gemini cost cap)"; `AI_SESSION_TURNS` no/`40`/"Session message cap — 2 per exchange; 10 = last 5 exchanges"). -- `DOC/POS.md:64`: parenthetical "(configurable via `AI_SESSION_TURNS`)" added after "capped at 40 turns" — recommended edit from design §Decision 4. -- `DOC/HOWTO.md:45`: `AI_MAX_TOKENS`, `AI_SESSION_TURNS` inserted after `AI_SYSTEM_PROMPT` in the `ai.env` var list. -- `DOC/AGENT_Context_Project.md:491`: same two vars added to the ai.env summary parenthetical. -- Line-count rows (hand-maintained, `:595-613`): no bump needed — none of those rows' files (install.sh, lib/*, features/*) changed length. `bin/pos-ai` row lives in the GENERATED filetable and was auto-updated 705 → 709 by `make gen`. - -[DONE] - -## Step 7: make gen x2 + make check + make lint - -- `make gen` ran 3× total (twice at implementation, once after the test suite): each rerun is a no-op → **idempotent**. -- Generated delta is ONLY the `bin/pos-ai` line-count row (705 → 709) in the filetable block. `completions/pos.bash` unchanged — verified `_pos_config_scopes` (`completions/pos.bash:55`) lists only scope *names* (already includes `ai`); individual keys are read live by `pos config` via `lib/config-ui.sh` from the `# POS_CONFIG:` headers (confirmed via `lib/registry.sh` `reg_scan bin` + `reg_config_keys ai` showing both new keys). The design's expectation that completions would gain the keys is a no-op in practice — no key-level completion table exists by design. -- `make check` (check-sync.sh): **OK** -- `make lint` (lint-conventions.sh): **0 FAIL, 0 WARN** - -[DONE] - -## Step 8: Smoke tests (no network — fake curl shim in /tmp/opencode/ai-cost-smoke) - -### Provider body cap (`test-provider-body.sh`, 12 checks, exit 0) - -Fake `curl` on PATH captures the `--data` body to a file and returns a canned 200 response + the `\n%{http_code}` line the adapters expect. - -| Case | Body assertion | Result | -|---|---|---| -| openrouter, AI_MAX_TOKENS unset | `"max_tokens":2048` | OK | -| openrouter, unset — JSON shape | model kept, messages kept | OK | -| openrouter, unset — canned response | `hello from fake openrouter` round-trips | OK | -| openrouter, AI_MAX_TOKENS=2048 | `"max_tokens":2048` | OK | -| openrouter, AI_MAX_TOKENS=512 | `"max_tokens":512` | OK | -| gemini, AI_MAX_TOKENS unset | `generationConfig.maxOutputTokens == 2048` | OK | -| gemini, unset — role conversion | assistant→model (unchanged behavior) | OK | -| gemini, unset — no systemInstruction when empty | absent | OK | -| gemini, unset — canned response | `hello from fake gemini` parses + round-trips | OK | -| gemini, AI_MAX_TOKENS=512 | `generationConfig.maxOutputTokens == 512` | OK | - -Canned bodies (shim): openrouter `{"choices":[{"message":{"role":"assistant","content":"hello from fake openrouter"}}]}`, gemini `{"candidates":[{"content":{"parts":[{"text":"hello from fake gemini"}]}}]}`; both + `200`. - -### session_push lazy resolution (`test-session-push.sh`, 10 checks, exit 0) - -`session_push` brace-extracted from `bin/pos-ai` (same `extract_fn` pattern as `tests/t-menu-allow-empty.sh`; function's braces are balanced so extraction is exact), with `MAX_SESSION_TURNS=40` declared as in `bin/pos-ai:25`. - -| Input | AI_SESSION_TURNS | Output length | Result | -|---|---|---|---| -| 30 msgs + 1 push | unset | 31 (no truncation under default) | OK | -| 30 msgs + 1 push | `10` | 10 (5 exchanges) | OK | -| 30 msgs + 1 push | `abc` (non-numeric) | 31 (fallback 40 → no truncation) | OK | -| 39 msgs + 1 push | unset | **40** (exact default boundary) | OK | -| 39 msgs + 1 push | `10` | 10 | OK | -| 39 msgs + 1 push | `0` (<1) | 40 (fallback) | OK | -| 39 msgs + 1 push | `-5` (negative) | 40 (fallback) | OK | -| 39 msgs + 1 push | `''` (empty) | 40 (fallback) | OK | - -Note: the brief's "30-message array → 40 default" cannot be exact (30+1=31 < 40, so nothing is trimmed); the 39+1 boundary case proves the exact default of 40. - -### Config registry discovery - -`reg_scan bin` + `reg_config_keys ai` via `lib/registry.sh` returns both new keys in the `ai` scope (`AI_MAX_TOKENS=num:…`, `AI_SESSION_TURNS=num:…`) — `pos config ai` will present them. - -### Full test suite - -`make test`: **17 files pass / 0 fail / 0 skip; 299 checks pass / 0 fail / 0 skip** (matches the 17/299 budget). - -### Other gates - -- `bash -n bin/pos-ai lib/ai-providers/openrouter.sh lib/ai-providers/gemini.sh` — clean -- `git diff --check` — clean -- `make gen` idempotent; `make check` OK; `make lint` 0 FAIL 0 WARN - -[DONE] - ---- - -## Files changed (diff summary) - -| File | Change | -|---|---| -| `bin/pos-ai` | POS_CONFIG header +2 keys; help Config block +2 lines; session_push lazy + guard (+4 net) | -| `lib/ai-providers/openrouter.sh` | body gains `max_tokens` | -| `lib/ai-providers/gemini.sh` | body gains `generationConfig.maxOutputTokens` | -| `DOC/POS.md` | table +2 rows; line 64 parenthetical | -| `DOC/HOWTO.md` | ai.env var list +2 | -| `DOC/AGENT_Context_Project.md` | ai.env parenthetical +2; generated filetable row auto-updated | - -## Scope compliance - -- In-scope changes only (design §Decision 5 file list). No llamacpp.sh, no pos-ai-server, no alias wrappers, no other tools, no tests, no AGENT_TODO.md edits. -- Provider call signature unchanged (`provider_generate "$model" "$messages" "$system"`). - -## Remaining risks / follow-up - -- `AI_MAX_TOKENS` too low truncates long `--full` answers (default 2048 mitigates; user can raise) — design's known risk. -- Env-vs-file precedence: exported env var beats config file (`load_env_file`) — expected and documented. -- Tester is recommended to add permanent provider-body + session-pruning regression tests (design §Testing). - -## Handoff - -Status: IMPLEMENTED -Recommended next agent: **Reviewer** (independent adversarial review of the diff before acceptance; Tester follows for permanent coverage per design §Testing). - ---- - -## Reviewer finding F1 fix (2026-09-06, after reviewer CHANGES_REQUIRED) - -**Source:** `AgentsReport/reviewer/2026-09-06_ai-cost-window-review.md` Finding 1 (REQUIRED) + Finding 2 (NOTE, folded). - -**Problem:** `AI_MAX_TOKENS=0`/`-5`/`010` are accepted by `pos config ai` (`num:` regex `^-?[0-9]+$`, config-ui.sh:419) and passed unguarded into jq `($mt|tonumber)` → provider 400 (0/-5) or raw jq abort (010 = invalid JSON literal; abc = tonumber error). The sibling `AI_SESSION_TURNS` guard `^[0-9]+$` also let `010` through (octal 8 ≥ 1) then aborted jq via `--argjson n "010"`. - -**Fix (exactly per reviewer demand, ~4 lines):** - -1. `lib/ai-providers/openrouter.sh:22-23`: - ```bash - local mt="${AI_MAX_TOKENS:-2048}" - [[ "$mt" =~ ^[1-9][0-9]*$ ]] || mt=2048 - ``` - then `--arg mt "$mt"` (previously `--arg mt "${AI_MAX_TOKENS:-2048}"`), keep `($mt|tonumber)`. -2. `lib/ai-providers/gemini.sh:17-18`: identical guard + use. -3. `bin/pos-ai:290-291` session guard tightened `^[0-9]+$` → `^[1-9][0-9]*$`; the `(( n >= 1 ))` check is now redundant (regex guarantees ≥ 1) and was dropped cleanly: - ```bash - local n="${AI_SESSION_TURNS:-$MAX_SESSION_TURNS}" - [[ "$n" =~ ^[1-9][0-9]*$ ]] || n="$MAX_SESSION_TURNS" - ``` - `--argjson n "$n"` is safe once n is strictly decimal (`[1-9][0-9]*` is a valid JSON numeric literal — no leading zeros). - -**Re-verification (same harness, extended cases):** - -| Check | Result | -|---|---| -| bash -n all three files | OK | -| Provider smoke (extended): AI_MAX_TOKENS unset / `0` / `-5` / `010` / `abc` → body `"max_tokens":2048`; `512` → `"max_tokens":512`; openrouter + gemini both | 16 checks OK | -| session_push smoke (extended): AI_SESSION_TURNS `0` / `010` / `-5` / `abc` → output length == default (31 for 30-array / 40 for 39-array boundary); `10` → 10 | 12 checks OK | -| `make gen` idempotent (POS_CONFIG unchanged; no new diff) | OK | -| `make check` | OK | -| `make lint` | 0 FAIL, 0 WARN | -| `make test` | 17 files / 299 checks green | -| `git diff --check` | clean | - -**Smoke matrix detail (provider body, openrouter + gemini):** - -| AI_MAX_TOKENS | openrouter body | gemini body | -|---|---|---| -| unset | `max_tokens:2048` | `maxOutputTokens:2048` | -| (empty) | `max_tokens:2048` | `maxOutputTokens:2048` | -| `0` | `max_tokens:2048` (fallback) | `maxOutputTokens:2048` (fallback) | -| `-5` | `max_tokens:2048` (fallback) | `maxOutputTokens:2048` (fallback) | -| `010` | `max_tokens:2048` (fallback, no jq abort) | `maxOutputTokens:2048` (fallback, no jq abort) | -| `abc` | `max_tokens:2048` (fallback, no jq abort) | `maxOutputTokens:2048` (fallback, no jq abort) | -| `512` | `max_tokens:512` | `maxOutputTokens:512` | - -**session_push matrix (30-msg array + 1 push):** - -| AI_SESSION_TURNS | output length | result | -|---|---|---| -| unset | 31 (all, default 40 no-op) | OK | -| `10` | 10 | OK | -| `0` | 31 (fallback 40) | OK | -| `010` | 31 (fallback 40 — former jq abort hole closed) | OK | -| `-5` | 31 (fallback 40) | OK | -| `abc` | 31 (fallback 40) | OK | - -**Scope compliance:** only the 3 already-approved files touched in this fix round; no design change, no jq logic shape change (still `($mt|tonumber)` / `generationConfig: {maxOutputTokens: …}`), no new files (report appended). - -[Handoff priority for F1 fix: Reviewer re-verify → Orchestrator gates → Tester permanent coverage; same recommendation as original handoff.] diff --git a/AgentsReport/builder/2026-09-06_ai-server-fixes.md b/AgentsReport/builder/2026-09-06_ai-server-fixes.md deleted file mode 100644 index 2e7c5b2..0000000 --- a/AgentsReport/builder/2026-09-06_ai-server-fixes.md +++ /dev/null @@ -1,103 +0,0 @@ -# Builder Report — llama-server breakage fixes (F1-F7) - -**Date:** 2026-09-06 -**Builder:** Implementation of Architect DQ1-DQ6 ratified scope F1-F7. -**Bindings:** `AgentsReport/architect/2026-09-06_ai-server-fix-design.md` (FACT), `AgentsReport/detective/2026-09-06_ai-server-breakage.md` (FACT root-cause evidence). - -## TL;DR - -Status: **IMPLEMENTED** (all F1-F7) - -Files changed: -- `bin/pos-ai-server` (F1, F2, F3, F4, F5, F6) -- `lib/common.sh` (F4 — new `ensure_user_bus`) -- `bin/pos-communication-matrix-listener` (F4 — inline + call) -- `bin/pos-network-download` (F4 — call) -- `apps/ai/llamacpp.sh` (F7 — post-install sanity) -- `DOC/POS.md` (minimal — `--no-unit` documented) -- generated docs via `make gen` (`DOC/AGENT_Context_Project.md`, `completions/pos.bash`) - -Verification: `bash -n` all touched; probes 1-8 = **63/63 PASS, rc=141 count 0**; `make gen` idempotent x2; `make check` green; `make lint` `0 FAIL, 0 WARN`. - -Probe workspace: `/tmp/opencode/ai-probes/` (stubs, fixtures, `run-probes.sh`, `PROBES-RESULT.txt`). - ---- - -## Step 1: F1 — `detect_llama_version` stderr capture + broadened regex - -`bin/pos-ai-server:82-90`. Real llama.cpp prints `version: 0.4.0-dev (build 10822, commit …)` to **STDERR**; the old code captured only stdout and regexed `[0-9]+\.[0-9]+\.[0-9]+` alone. - -- Both streams captured: `"$bin" --version 2>&1`. -- Regex accepts `X.Y.Z` OR `build [0-9]+` OR `b[0-9]+` (old builds). -- `"${version#build }"` strips the phrase so `build 10822` displays as `10822`. -- Guarded: missing binary / unreadable output → `unknown` (no errexit). - -Evidence (P1): stderr semver → `0.4.0`; `build 10822` (stderr) → `10822`; `b10822` → `b10822`; missing binary → `unknown`. [DONE] - -## Step 2: F2 — pipe-less flag validation (no rc=141 race) - -`bin/pos-ai-server:97-124` (`validate_requested_flags`), `:135-170` (`validate_default_flags`). Old form `printf '%s' "$help_text" | grep -qE ...` died of SIGPIPE (rc=141) when the >64KB pipe-adjacent grep exited at the first match — valid flags sporadically judged unsupported. New form: non-quiet `grep -E -- "(^|[[:space:]])${flag}([[:space:]]|=|$)" <<<"$help_text" >/dev/null` (herestring needs no pipe; non-q grep consumes the whole input; same word-boundary regex and guard shape). - -Evidence (P2): on the 58,362-byte/732-line fixture help, 25 runs of `validate_default_flags` → all 5 defaults kept every run, `rc=141` count **0**; requested-flag path: supported `--port` rc 0, unsupported flag rc 1 with version-aware err. [DONE] - -## Step 3: F3 — `resolve_model` directory expansion (never silently pick) - -`bin/pos-ai-server:213-264` + helper `resolve_gguf_in_dir` (`:196-210`). DQ2 precedence: absolute file → absolute dir with exactly one `.gguf` → `$HF_DOWNLOAD_DIR//.gguf` (slug/file form) → `$HF_DOWNLOAD_DIR/.gguf` (flat file) → relative-as-is. A dir with multiple `.gguf`s errors listing each as `/` + "pick one"; zero `.gguf`s falls through to the today's `Model not found:` err. - -Evidence (P3): single-gguf dir resolves; slug/file form resolves; multi-gguf errors listing both files + "pick one"; empty dir → rc 1 with today's `Model not found: empty-dir` text. [DONE] - -## Step 4: F4 — `ensure_user_bus` + `--no-unit` escape hatch - -`lib/common.sh` (new `ensure_user_bus`, after `confirm()`): `systemctl --user show-environment` reachability probe; on failure `err` with BOTH remediation lines (`export XDG_RUNTIME_DIR=/run/user/$(id -u)` and `sudo loginctl enable-linger $(id -un)`) + "no unit was written". Called: - -- `bin/pos-ai-server` `cmd_start` (before DRY_RUN return; only when `--no-unit` not given). `--no-unit` flag: added to `# POS_FLAGS:`, parse case, usage; direct run via `eval "nohup $exec_cmd >'$NO_UNIT_LOG' 2>&1 &"`, pidfile/log under `${XDG_RUNTIME_DIR:-/tmp}`; `cmd_stop`/`cmd_status` pidfile-aware (`kill -0`, `^[0-9]+$` guard). -- `bin/pos-communication-matrix-listener` — **factually does NOT source `lib/common.sh`** (inline `err`/`log`/`warn` at lines 19-21); got an inline `declare -F ensure_user_bus`-guarded duplicate + call in `enable_service` (architect's "all three source common.sh" premise was wrong; sourcing it would redefine helpers with colors and change output). -- `bin/pos-network-download` `cmd_start` else-branch (first statement before unit write). - -Evidence (P4): with `XDG_RUNTIME_DIR`/`DBUS_SESSION_BUS_ADDRESS` unset and a failing `systemctl` stub: pos-ai-server DRY_RUN start rc 1 with both remediation lines and **no unit written**; matrix-listener `--enable` rc 1, no unit; network-download `start` rc 1, no unit. [DONE] - -## Step 5: F5 — `find_llamacpp` candidate narrowing - -`bin/pos-ai-server:56-61`. Candidates narrowed to `("llama-server" "llama-server-cuda")` (dropped bare `server` and `llama.cpp/server` which had previously matched unrelated binaries). Stale-unit overwrite warn added in start; user unit state never deleted. - -Evidence (P5): PATH containing only an unrelated binary named `server` → `find_llamacpp` returns nothing, rc 1; stub `llama-server` → rc 0, returns `llama-server`. [DONE] - -## Step 6: F6 — port pinning preserved (no change needed, verified) - -Verified via P6: DRY_RUN start with the b10822-shaped stub help → ExecStart contains all 5 defaults (`--host 127.0.0.1`, `--port 8088`, `--n-gpu-layers`, `--ctx-size 4096`, `--threads`) and the version resolves to `0.4.0` (not `unknown`). [DONE] - -## Step 7: F7 — llamacpp.sh post-install sanity - -`apps/ai/llamacpp.sh`: new `llamacpp_sanity()` (seam `LLAMACPP_BIN_DIR="${LLAMACPP_BIN_DIR:-/usr/local/bin}"`), called after install `spawn`. Verifies: `[ -e "$bin" ]` (catches dangling symlink), `command -v llama-server`, `llama-server --version >/dev/null 2>&1`, `llama-server --help >/dev/null 2>&1`; on success logs `llama.cpp sanity OK`. - -Evidence (P7): normal fixture → rc 0 + OK log; dangling-symlink fixture → rc 1 with "dangling symlink"; missing-shared-lib fixture → rc 1 with the binary's stderr text. [DONE] - -## Step 8: E2E user scenario - -Evidence (P8): positive (bus env set, `systemctl` stub succeeds): `pos ai-server start Qwen-Qwen3-1.7B-GGUF` rc 0, unit written with `--port 8088`, all 4 other defaults, model path expanded from dir, **no "unknown" version anywhere in start output**. SSH-shaped (no bus env, failing stub): rc 1 with both remediation lines, **no orphan unit**. [DONE] - -## Gates - -| Gate | Result | -|---|---| -| `bash -n` on 5 touched scripts | pass (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh) | -| `make gen` x2 | rc 0 both; `git diff` identical → **idempotent** | -| `make check` | `check-sync: OK` | -| `make lint` | `0 FAIL, 0 WARN (convention lint)` | -| Probes 1-8 | **63 PASS / 0 FAIL / rc=141 count 0** | - -## Scope compliance - -- Touched exactly the in-scope files (pos-ai-server, common.sh, matrix-listener, network-download, llamacpp.sh, DOC/POS.md, gen output). `tests/`, `scripts/lint-conventions.sh`, `bin/pos-system-uninstall`, `lib/config-ui.sh` untouched. -- Unrelated pre-existing dirty file `AgentsReport/architect/2026-09-06_stabilization-design.md` NOT modified by Builder. -- No commits made. - -## Residual risks / known deviations - -1. `--no-unit` direct run uses `eval "nohup $exec_cmd … &"` — tool-generated string; only binary (validated by `command -v`) and model (validated path) tokens are interpolated, both quoted; matches the namespace's bash DNA. -2. F1 displays `build 10822` as `10822` via `"${version#build }"` — slight deviation from the architect's literal regex output wording, required to match probe 1's `10822` expectation; behavior documented at `bin/pos-ai-server:75-81`. -3. `matrix-listener` carries an inline duplicate of `ensure_user_bus` (guarded by `declare -F` so the shared one wins if ever sourced) — the file deliberately does not source common.sh (would change helper output). If a future task refactors matrix-listener to source common.sh, the inline copy can be deleted. -4. E2E positive run writes/overwrites a unit under `$USER_SYSTEMD_DIR` only inside the probe workspace — no real system units touched, no processes spawned (stubs only). -5. Runtime flags validated by grep use `>`-redirected non-quiet grep instead of `grep -q`: a 58KB fixture costs ~1-2ms per flag — negligible for a unit-write path. - -Recommended next agent: **Reviewer** (independent adversarial review of the F1-F7 diff before acceptance). \ No newline at end of file diff --git a/AgentsReport/builder/2026-09-06_alias-menu-fix.md b/AgentsReport/builder/2026-09-06_alias-menu-fix.md deleted file mode 100644 index a2ded70..0000000 --- a/AgentsReport/builder/2026-09-06_alias-menu-fix.md +++ /dev/null @@ -1,49 +0,0 @@ -# Builder Report — Alias Menu Fix (2026-09-06) - -## TL;DR -- **Status:** IMPLEMENTED -- **Files changed:** `lib/menu-lib.sh`, `bin/pos-ai-alias`, `AgentsReport/builder/2026-09-06_alias-menu-fix.md` -- **Verification:** bash -n OK; make gen ×2 (deterministic, no drift); make check OK; make lint 0 FAIL 0 WARN; functional smoke 7/7 pass. -- **Summary:** Added `--allow-empty` flag to `menu_ask_value`, switched 2 alias call sites (353, 410), fixed step counters 352/383 → /5. - ---- - -## Step 1: Update function index line in lib/menu-lib.sh -Updated line 27: `menu_ask_value