Linux_post_install/AgentsReport/architect/2026-09-06_ai-cost-window-design.md

# AI cost & session-window design — OpenRouter 402 + bounded session memory

**Date:** 2026-09-06
**Author:** Architect (big-pickle)
**HEAD:** 8ce5479 (clean tree)
**Mode:** design-only (no file edits besides this report)

## TL;DR

Two tightly associated defects in `pos ai` / `bin/pos-ai`:

1. **402 root cause** — no provider ever sends `max_tokens`, so OpenRouter's credit
   pre-check charges the routed model's full worst-case output (131072 on
   `openrouter/auto`) and rejects balances under that. Fix: send an explicit capped
   `max_tokens`.
2. **Unbounded-ish session** — `MAX_SESSION_TURNS=40` counts *messages* (20
   exchanges); user wants "last 5 req/response" = 10 messages.

**Decisions:** add `AI_MAX_TOKENS` (num, default **2048**) honored by openrouter
(body `max_tokens`) and gemini (`generationConfig.maxOutputTokens`); **skip** llamacpp.
Add `AI_SESSION_TURNS` (num, default **40** — backward compatible) honored lazily in
`session_push`; user sets **10** for 5 pairs. Both vars declared in the `@General`
section of the `# POS_CONFIG:` header, docs updated. No chat/alias wrapper changes.

**Open items:** none blocking. Tester should add provider-body + session-pruning
coverage (see §Testing).

---

## Verified fact confirmation (A–D)

All user-reported facts confirmed against source at HEAD:

**A — 402 root cause: CONFIRMED.**
- `lib/ai-providers/openrouter.sh:22-23` — body is only `{model,messages}`; no
  `max_tokens`. `:24-29` POSTs straight to OpenRouter with unchanged body.
- `lib/ai-providers/gemini.sh:17-19` — body only `{contents,...}`; no
  `generationConfig`. (`:20-23` adds only `systemInstruction`.)
- `lib/ai-providers/llamacpp.sh:32-33` — body `{model,messages,stream:false}`.
- Conclusion: none carry a generation cap → OpenRouter 402 with the user's thin
  balance. Fix is to send an explicit `max_tokens`.

**B — session window: CONFIRMED.**
- `bin/pos-ai:25` `MAX_SESSION_TURNS=40`.
- `session_push()` `bin/pos-ai:285-290` → `'.messages |= .[-"$MAX_SESSION_TURNS":]'`
  prunes to last N **messages** (40 msgs = 20 exchanges). User wants 5 pairs = 10 messages.

**C — config surface: CONFIRMED.**
- `# POS_CONFIG:` header `bin/pos-ai:6`, scope `ai | ai.env`.
- `num:` type already used in the same header (`LLAMACPP_CTX_SIZE=num:…`,
  `LLAMACPP_GPU_LAYERS=num:…`).
- Validation: `lib/config-ui.sh:419` `*,num,*) [[ "$val" =~ ^-?[0-9]+$ ]]` — integer-only
  on *entry*; empty input ="kept current value" (`:468-471`); `-` = clear (`:472-477`).
  So `num:` + empty/unset → falls back to code default. Clean.
- Env precedence: `load_env_file` (`lib/config-ui.sh:336-357`) exports a file key only
  when the variable is not already set in the environment (`:351-353`) → env beats
  file beats default. Providers read config via env (`AI_API_KEY` pattern).
- `PROVIDER_CONFIG` headers exist on `openrouter.sh:7-8`, `gemini.sh:7-8`,
  `llamacpp.sh:7` for *provider-specific* keys.

**D — provider resolution: CONFIRMED.**
- `bin/pos-ai:679-683` — `--provider` flag > `AI_PROVIDER` env > `gemini`.
- `load_config()` `bin/pos-ai:141-151` loads `ai.env` (+ legacy files); called by
  `resolve_key` (line 155) and at provider resolution (line 681).
- Call site `bin/pos-ai:522` (ask) and `:557` (chat): `provider_generate "$model" "$messages" "$system"`.

**Gates/tests scan:**
- No test pins `MAX_SESSION_TURNS`, the provider request bodies, or header text
  (`t-config-precedence.sh` targets `pos-ai-server`, a separate tool). No forced
  test update.
- `DOC/POS.md` has a hand-maintained ai.env config table (`:88-97`) and mentions
  "capped at 40 turns" at `:64`. `DOC/HOWTO.md:45` and `DOC/AGENT_Context_Project.md:491`
  list ai.env vars (hand-maintained). All need doc rows/bumps for the new vars.

---

## Decision 1: `AI_MAX_TOKENS` — cap generation tokens

**Status: [DECIDED]**

### Options & trade-offs

**Option 1 (chosen) — single global `AI_MAX_TOKENS=num`, default 2048, honored by remote providers (openrouter + gemini); skip llamacpp.**
- *Advantages:* smallest change that fixes the 402 (OpenRouter pre-check sees a
  capped cost) and is a real per-request cost ceiling; one var, one default; fits
  the existing `AI_*` env naming and the tool-level `@General` config section; no
  new per-provider surface.
- *Costs:* remote providers share one ceiling (no per-provider cap without user
  intervention).
- *Risks:* a too-low cap truncates long answers — mitigated by default 2048 being
  ample for terse `ask`/`chat` CLI answers; user can raise it.
- *Reasoning for default 2048:* conservative (user's balance affords ~4511 tokens at
  routed price, so 2048 passes the pre-check with margin) while being a practical,
  real ceiling. 2048 tokens ≈ several thousand chars — plenty for the terse,
  commands-first assistant role this tool plays.

**Option 2 — per-provider caps via `PROVIDER_CONFIG` (e.g. `AI_MAX_TOKENS` on openrouter.sh, gemini.sh).**
- *Advantages:* independent ceilings per provider.
- *Costs:* two declarations, redundant section plumbing, and the *default* (which is
  the entire point) still has no shared home → awkward. `PROVIDER_CONFIG` is for
  provider-specific concerns; a cost ceiling + 402 pre-check over both remote
  providers is tool-level, not provider-specific.

**Option 3 — no gemini cap; only openrouter.**
- *Advantages:* minimal (402 only affects OpenRouter).
- *Costs:* leaves Gemini without any cost ceiling while introducing the same var —
  inconsistent, and Gemini's own pricing can surprise. Rejected.

**Option 4 — include llamacpp too (`max_tokens` in body; it accepts it).**
- *Advantages:* provider parity on the OpenAI-compatible endpoint.
- *Costs:* local & free — no credit pre-check, no cost. Adds surface with zero user
  benefit. Rejected on the "smallest sufficient design" principle.

### Implementation contract
- Env var name: **`AI_MAX_TOKENS`**, type `num`, default **2048**.
- OpenRouter body (`openrouter.sh:22-23`): add `max_tokens`.
- Gemini body (`gemini.sh:17-19`): add `generationConfig.maxOutputTokens`.
- llamacpp: **no change**.
- Providers read `"${AI_MAX_TOKENS:-2048}"` from env; value is present because
  `load_config` runs before `provider_generate` (resolve path confirmed in D).
- Declare in `@General` section of `# POS_CONFIG:` header (`bin/pos-ai:6`).

---

## Decision 2: `AI_SESSION_TURNS` — bounded session window

**Status: [DECIDED]**

### Options & trade-offs

**Option 1 (chosen) — new `AI_SESSION_TURNS=num`, default 40 (unchanged), resolved lazily in `session_push`.**
- *Advantages:* fully backward compatible — no silent memory truncation for existing
  users. The user's "last 5 req/response" = setting `AI_SESSION_TURNS=10`. One var,
  one default.
- *Costs:* existing users must opt in (they already have the 40 behavior, so no
  regression).
- *Why keep default 40:* backward compatibility is a hard project value; silently
  shifting the default changes session context for *every* user, discards history
  they may rely on, and is a behavioral change not requested globally (only for this
  user). Keep 40.

**Option 2 — change the default to 10.**
- *Advantages:* meets the stated want out-of-the-box.
- *Costs:* silent behavior change for all users; discards memory; not requested
  globally. Rejected — keep the change opt-in via the new var.

### Semantics (must be documented)
The existing prune is `.messages |= .[-N:]` where N counts **messages** — 2 messages
per exchange. So `AI_SESSION_TURNS=10` ⇒ last **5** exchanges (5 user + 5 assistant).
The config description must state: "message count (2 per exchange); 10 = last 5
exchanges".

### Implementation constraint — lazy resolution (important)
`MAX_SESSION_TURNS` is currently assigned at `bin/pos-ai:25`, which executes at
top-level **before** `load_config` is first called (line 681). If we wrote
`MAX_SESSION_TURNS="${AI_SESSION_TURNS:-40}"` at line 25, an `AI_SESSION_TURNS` set
*only in ai.env* would not yet be loaded → always 40.

Therefore:
- Keep **line 25** as-is (`MAX_SESSION_TURNS=40`), used for the help text (`:67`
  shows the default, accurate).
- In `session_push()` (lines 285-290), resolve **lazily**:
  `local n="${AI_SESSION_TURNS:-$MAX_SESSION_TURNS}"` and use `$n` in the jq prune.
  Because `session_push` runs inside `cmd_ask`/`cmd_chat` — after `load_config`
  (via `resolve_key` at `:499`/`:538`) has exported `AI_SESSION_TURNS` into the
  process env — the ai.env value is honored. Shell-exported `AI_SESSION_TURNS`
  wins too (env-wins in `load_env_file`).
- Help text `bin/pos-ai:67` stays accurate ("capped at $MAX_SESSION_TURNS turns")
  since the default remains 40. Optionally add a `Config:` help line documenting the
  var — recommended, and it confirms "turns = messages, 10 = 5 pairs".

---

## Decision 3: `# POS_CONFIG:` header change

**Status: [DECIDED]**

- Single `@General` section addition (not per-provider `PROVIDER_CONFIG`):
  - `AI_MAX_TOKENS=num:Max output tokens per request (default 2048; OpenRouter/Gemini cost cap)`
  - `AI_SESSION_TURNS=num:Session message cap — 2 per exchange (default 40 = 20 exchanges; 10 = last 5)`
- Place both in the `@General` section alongside `AI_SYSTEM_PROMPT` (end of the
  long header line, `bin/pos-ai:6`).
- **Why @General, not PROVIDER_CONFIG:** the default is shared across providers
  (2048, 40) and both caps are tool-level concerns. `PROVIDER_CONFIG` is reserved for
  provider-specific keys (API keys, models). A single global declaration is the
  cleanest and avoids duplicating the default in two provider files.
- `num:` type confirmed suitable: `lib/config-ui.sh:419` enforces integer on entry;
  empty=keep current, `-`=clear (`:468-477`); unset → code default. No empty-parse
  concern.
- Provider bodies consume the vars from env, so no `PROVIDER_CONFIG` additions are
  needed on `openrouter.sh`/`gemini.sh`. (They could be added later if per-provider
  caps are ever wanted — out of scope now.)

---

## Decision 4: Docs & gates

**Status: [DECIDED]**

- **`DOC/POS.md`** (hand-maintained):
  - ai.env table (`:88-97`): add rows for `AI_MAX_TOKENS` (no/`2048`/"Max output
    tokens per request (OpenRouter/Gemini cost cap)") and `AI_SESSION_TURNS`
    (no/`40`/"Session message cap — 2 per exchange; 10 = last 5 exchanges"). The
    `AI_SYSTEM_PROMPT` row (`:93`) is the placement anchor.
  - Line 64 text "capped at 40 turns" remains true (default unchanged) — no edit
    strictly needed, but a short parenthetical "(configurable via AI_SESSION_TURNS)"
    is recommended.
- **`DOC/HOWTO.md:45`** — append `AI_MAX_TOKENS`, `AI_SESSION_TURNS` to the listed
  ai.env vars.
- **`DOC/AGENT_Context_Project.md:491`** — append the two vars to the ai.env
  summary parenthetical (hand-maintained).
- **`make gen`**: header text change does not add commands/subcommands/flags, so the
  generated tree/dispatch tables are unaffected; `completions/pos.bash` config-scope
  table regenerates to include the new keys. Run `make gen` (deterministic, `LC_ALL=C`
  per convention), then `make check`, then `make lint`.
- **Line-count rows** above the filetable marker in `AGENT_Context_Project.md`: only
  bump a row if a `pos-*`/`lib/*` file's length changes (it does — lib/ai-providers
  grow; bin/pos-ai grows). Do not touch rows for files that don't change.
- **tests**: existing suite does not pin the session default or provider bodies, so
  nothing is *forced*. Recommend new coverage (§Testing).

---

## Decision 5: Scope fence

**Status: [DECIDED]**

**Approved outcome:** OpenRouter 402 eliminated (explicit capped `max_tokens` on
remote providers) and session history bounded via configurable `AI_SESSION_TURNS`.

**In-scope files:**
- `bin/pos-ai` — `# POS_CONFIG:` header (line 6, @General additions); `session_push`
  lazy `AI_SESSION_TURNS` resolution (lines 285-290); optional `Config:` help lines
  for the two vars. Line 25 stays `MAX_SESSION_TURNS=40`.
- `lib/ai-providers/openrouter.sh` — add `max_tokens` to body (lines 22-23).
- `lib/ai-providers/gemini.sh` — add `generationConfig.maxOutputTokens` (lines 17-19).
- `lib/ai-providers/llamacpp.sh` — **no change**.
- Docs (hand-maintained): `DOC/POS.md`, `DOC/HOWTO.md`, `DOC/AGENT_Context_Project.md`.
- tests (later, Tester).

**Allowed interface changes:** two new config keys in scope `ai`; provider request
bodies gain a token cap. Provider call signature `provider_generate "$model" "$messages" "$system"`
is unchanged.

**Explicitly out of scope:** `bin/pos-ai-server`, chat/alias wrappers, `pos-ai-alias`,
HuggingFace downloader, other tools, per-provider caps, changing `MAX_SESSION_TURNS`
default, any llmacpp body change.

**Architectural constraints:**
- No new provider-side config plumbing; vars read from env (`AI_*` convention).
- Backward compatible: session default 40 and empty/unset values fall back to
  documented defaults.
- Deterministic `make gen`; `make check` + `make lint` green (definition of done).

**Open risks:**
- `AI_MAX_TOKENS` too low truncates long `--full` answers — default 2048 mitigates;
  user can raise.
- env-vs-file precedence: exported env var beats config file (per `load_env_file`) —
  expected and documented.

**Verification:**
- `bash -n` clean; `make gen`/`make check`/`make lint` green.
- A 402 reproduction no longer triggers when `AI_MAX_TOKENS` is set/at default.
- `AI_SESSION_TURNS=10` prunes the session to last 5 exchanges.

---

## Testing budget (suggestion for Tester / Builder verification)

- **Provider body cap:** source each provider adapter in a sandbox with `curl`
  stubbed, assert the request JSON contains `max_tokens` (openrouter) / a truthy
  `generationConfig.maxOutputTokens` (gemini). Pattern: `t-config-precedence.sh` style
  curl-log capture with a fake `curl` in `PATH` (see existing tests).
- **Session pruning:** drive `session_push` with a crafted messages JSON and
  `AI_SESSION_TURNS=10`, assert exactly the last 10 messages remain (5 pairs); and
  `AI_SESSION_TURNS` unset → 40 retained (default commits to backward compat).
- **Help/default fidelity:** `pos ai --help` still shows 40; `pos config ai` lists the
  two new `num:` keys and rejects a non-integer (`cfg_validate`).
- **Regression:** existing `t-config-precedence.sh`, `run-tests` full suite green.

---

## Handoff

**Status:** DECISION_READY

**Problem:** OpenRouter 402 (no `max_tokens` → full worst-case pre-check) and
unbounded session memory; user wants 5 req/response.

**Decision:** add `AI_MAX_TOKENS=num` (default 2048) honored by openrouter + gemini
(skip llamacpp); add `AI_SESSION_TURNS=num` (default 40, backward compatible) resolved
lazily in `session_push`; both in `@General` config section; docs updated.

**Ownership:** `bin/pos-ai` + `lib/ai-providers/{openrouter,gemini}.sh` (+ docs).

**Interfaces:** two new `ai`-scope config keys; provider bodies gain a token cap.
Call signature unchanged.

**Approved scope / constraints / verification / out-of-scope:** see Decision 5.

**Risks:** see Decision 5 (token-cap truncation; env-vs-file precedence — all mitigated).

**Recommended next agent:** **Builder**

**Reason:** the architecture and scope are fully specified with exact line-level
changes (no architectural ambiguity left). Builder can implement without making
architecture decisions. Tester follows for the recommended coverage.