feat: pos ai hf — Hugging Face model downloader for local inference
gates / consistency-and-conventions (push) Successful in 1m32s
gates / consistency-and-conventions (push) Successful in 1m32s
Bash-native tool using curl/jq to download AI models from HF Hub. Subcommands: download (single file/repo/gguf filter), search, list, remove. Auth via HF_TOKEN in ai.env, resume support, disk space pre-flight, rate limit handling, .hf-meta metadata tracking. 46 test cases / 0 failed. make gen/check/lint 0 FAIL / 0 WARN.
This commit is contained in:
@@ -42,6 +42,8 @@ summary (newest last).
|
||||
|
||||
## Done
|
||||
|
||||
- **2026-09-04** — `pos ai hf` (`bin/pos-ai-hf`) — Hugging Face model downloader. Subcommands: `download <repo-id> [filename]` (single file, whole repo, `--gguf` filter, `--branch <rev>`, `--output <dir>`), `search <query>`, `list`, `remove`. Downloads to `~/.local/share/linux_post_install/ai/models/<namespace>-<model-name>/` (seam-guarded `HF_DOWNLOAD_DIR`), writes `.hf-meta` JSON per repo, prints structured summary (📥/📁). Config extends the existing `ai` scope via `# POS_CONFIG: ai` — `HF_TOKEN` (secret) and `HF_DOWNLOAD_DIR` in `~/.config/linux_post_install/ai.env` with env-var precedence. Auth on all requests; HTTP 429 rate-limit sleep + retry once; resume via `curl -C -`; progress bars to stderr. Deps: `curl`/`jq` guards before `--help`; no stdin → not in INTERACTIVE_CMDS. Verified: stub-PATH suite `/tmp/opencode/hf-test/run-tests.sh` 46/46 green (argument parsing, download single/multi/gguf/branch/output, search, list, remove, config/token, output format); `bash -n`; `make gen && make check` green; `make lint` 0 FAIL / 0 WARN. Docs: POS.md ai row + detail block.
|
||||
|
||||
- **2026-09-04** — `pos media grab` (`bin/pos-media-grab`) — auto-download a URL as audio or video. Classifies by domain (YouTube Music/SoundCloud/Bandcamp → mp3; YouTube/Vimeo/Twitch → mp4) with `--audio`/`--video` overrides and `GRAB_DEFAULT` config (`pos config grab`, default `video`) for unknown domains; `--best` default for video (non-interactive, `--worst` override); all flags (`--output`, `--no-playlist`, `--cookies`, `--dry-run`) forwarded to mp3/mp4; prints a clean summary (🎵/🎬 title, duration, path, size). Telegram listener (`bin/pos-communication-telegram-listener`) gains `url_detect` + a URL routing step between the prefix map and AI bridge — bare http(s) URLs route to `pos media grab --best` (600s timeout). Verified: `/tmp/opencode/media-grab-test/run-tests.sh` 28 cases / 70 assertions green; `bash -n` on both files; `make gen && make check` green; `make lint` 0 FAIL / 0 WARN.
|
||||
|
||||
- **2026-08-21** — Share suite interactive layer (`lib/share-lib.sh` + menu modes for all five `pos share *` tools): bare invocation now opens an EOF-safe looping menu instead of printing usage. New `lib/share-lib.sh` (436 lines) owns the shared primitives — `share_menu_guard`/`share_menu_run`/`share_pick`/`share_ask_value` (quit on EOF so non-tty callers can't hang), `share_require_bin`/`share_port_probe`/`share_service_active`/`share_path_probe` rc-only probes, `share_usb_records` (blank-line-record parser for usbsrv listings), `share_smb_shares` (smbclient `-g` Disk enumeration incl. guest→auth retry) + `share_usb_devices`/`share_usb_clients`, `share_nfs_exports` (showmount), `share_folder_candidates` (bounded-probe scan of mounted targets + conventional roots; container overlay/tmpfs/nsfs excluded via findmnt; clients build their own mountpoint pickers on top), and advisories `share_ufw_blocks_ports`+`share_offer_fix`. Tools keep every legacy flag/subcommand byte-compatible (verbatim command bodies, thin menu layer on top): nfs-server gains a client-spec presets picker + inactive-service/UFW offers, nfs-client gains idempotent unmount/unpersist (already-absent = report, rc0) + persist-verify-with-rollback + replace-confirm, smb-server gets UFW offer + menu tree, smb-client gets enumerate→pick→mount with account reuse (`SMB_AUTH_USER` contract), usb-server picker-first with raw-listing manual-entry fallback when the server listing is unreadable. New seams: `EXPORTS_FILE` (nfs-server), `UNIT_DIR` (nfs-client); `bin/pos` INTERACTIVE_CMDS += both stdin-reading share tools; install.sh lib list += share-lib.sh; preinstall.sh += smbclient (smb-client enumeration dep). Docs: POS.md share rows/detail, howto/share.md per-tool Interactive-menu notes, SCRIPTS.md phase table + new `## lib/share-lib.sh` section, DEV.md lib row + env-seam registry, AGENT_Context hand-maintained spots (lib table row 436, Phase-2 prose). Verified: 80-case stub battery vs recaptured deterministic golden — only the 9 documented intentional deltas differ (additive help lines, seam-path strings in messages, missing-dep message delta, unmount-idle FLAGGED→rc0, unpersist round-trip now works, usbs-bare usage→menu guard); 12/12 PTY tests (menus open/quit non-tty, filter/zero-match/default/cancel picker semantics, full nfs-server share flow writes the export line, usb-server share via pickers + down-server fallback, smb-client guest-enumerate→manual-share flow); real `/etc/exports` + `/etc/systemd/system` md5-verified untouched; `make gen && make check` green, `make lint` 0 FAIL / 0 WARN.
|
||||
|
||||
@@ -10,19 +10,19 @@
|
||||
|
||||
<!-- GEN:START docmap -->
|
||||
| ## 1. Project Overview | 28–43 |
|
||||
| ## 2. Directory Structure | 44–205 |
|
||||
| ## 3. Installation Flow | 206–259 |
|
||||
| ## 4. The `pos` CLI System | 260–338 |
|
||||
| ## 5. Shared Library — `lib/common.sh` | 339–370 |
|
||||
| ## 6. Docker Compose / ScaleTail | 371–413 |
|
||||
| ## 7. Optional Apps (`apps/`) | 414–443 |
|
||||
| ## 8. Entertainment Module | 444–457 |
|
||||
| ## 9. Systemd Services | 458–469 |
|
||||
| ## 10. Configuration Files | 470–496 |
|
||||
| ## 11. Coding Conventions | 497–529 |
|
||||
| ## 12. Development Workflow | 530–582 |
|
||||
| ## 13. Key File Quick Reference | 583–655 |
|
||||
| ## 14. Common Tasks for Agents | 656–689 |
|
||||
| ## 2. Directory Structure | 44–207 |
|
||||
| ## 3. Installation Flow | 208–261 |
|
||||
| ## 4. The `pos` CLI System | 262–341 |
|
||||
| ## 5. Shared Library — `lib/common.sh` | 342–373 |
|
||||
| ## 6. Docker Compose / ScaleTail | 374–416 |
|
||||
| ## 7. Optional Apps (`apps/`) | 417–446 |
|
||||
| ## 8. Entertainment Module | 447–460 |
|
||||
| ## 9. Systemd Services | 461–472 |
|
||||
| ## 10. Configuration Files | 473–499 |
|
||||
| ## 11. Coding Conventions | 500–532 |
|
||||
| ## 12. Development Workflow | 533–585 |
|
||||
| ## 13. Key File Quick Reference | 586–659 |
|
||||
| ## 14. Common Tasks for Agents | 660–693 |
|
||||
<!-- GEN:END docmap -->
|
||||
|
||||
## 1. Project Overview
|
||||
@@ -63,6 +63,8 @@ Linux_post_install/
|
||||
<!-- GEN:START tree -->
|
||||
│ ├── pos-ai-alias # manage AI agent aliases
|
||||
│ ├── pos-ai-gemini # Forward to pos ai --provider gemini (backward compat)
|
||||
│ ├── pos-ai-hf # Download AI models from Hugging Face (search, download, manage)
|
||||
│ │ [deps: curl jq]
|
||||
│ ├── pos-ai-openrouter # Forward to pos ai --provider openrouter (backward compat)
|
||||
│ ├── pos-communication-matrix-listener # Matrix listener: map /command → bash, run them on room messages
|
||||
│ ├── pos-communication-matrix-sender # Send messages to a Matrix room via the client-server API (send, test, login)
|
||||
@@ -278,6 +280,7 @@ All non-interactive `pos` commands log output to `~/.local/share/linux_post_inst
|
||||
|----------|---------|--------|-------------|------|----------|
|
||||
| ai | alias | `pos-ai-alias` | manage AI agent aliases | | |
|
||||
| ai | gemini | `pos-ai-gemini` | Forward to pos ai --provider gemini (backward compat) | | |
|
||||
| ai | hf | `pos-ai-hf` | Download AI models from Hugging Face (search, download, manage) | curl jq | pos ai hf search llama 7b → Search Hugging Face for "llama 7b" models · pos ai hf download meta-llama/Llama-3.1-8B-Instruct → Download all files from a repo · pos ai hf download meta-llama/Llama-3.1-8B-Instruct --gguf → Download only GGUF quantized files · pos ai hf download meta-llama/Llama-3.1-8B-Instruct config.json → Download a single file · pos ai hf list → List downloaded models · pos ai hf remove meta-llama-Llama-3.1-8B-Instruct → Remove a downloaded model |
|
||||
| ai | openrouter | `pos-ai-openrouter` | Forward to pos ai --provider openrouter (backward compat) | | |
|
||||
| communication | matrix-listener | `pos-communication-matrix-listener` | Matrix listener: map /command → bash, run them on room messages | | |
|
||||
| communication | matrix-sender | `pos-communication-matrix-sender` | Send messages to a Matrix room via the client-server API (send, test, login) | | |
|
||||
@@ -607,6 +610,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos` | 302 | CLI dispatcher with smart arg matching + logging + category help |
|
||||
| `bin/pos-ai-alias` | 760 | manage AI agent aliases |
|
||||
| `bin/pos-ai-gemini` | 7 | Forward to pos ai --provider gemini (backward compat) |
|
||||
| `bin/pos-ai-hf` | 495 | Download AI models from Hugging Face (search, download, manage) |
|
||||
| `bin/pos-ai-openrouter` | 7 | Forward to pos ai --provider openrouter (backward compat) |
|
||||
| `bin/pos-communication-matrix-listener` | 568 | Matrix listener: map /command → bash, run them on room messages |
|
||||
| `bin/pos-communication-matrix-sender` | 224 | Send messages to a Matrix room via the client-server API (send, test, login) |
|
||||
@@ -647,7 +651,7 @@ Use conventional prefixes: `feat:`, `fix:`, `docs:`, `refactor:`, `chore:`
|
||||
| `bin/pos-ai` | 692 | AI assistant: ask, chat, sessions, capture, models, providers |
|
||||
| `bin/pos-config` | 80 | Interactive editor for the tools' runtime config (reads # POS_CONFIG: registry) |
|
||||
| `bin/pos-tree` | 118 | Show the pos CLI command tree: categories, commands, and subcommands |
|
||||
| `completions/pos.bash` | 309 | Dynamic bash completion |
|
||||
| `completions/pos.bash` | 310 | Dynamic bash completion |
|
||||
<!-- GEN:END filetable -->
|
||||
| `apps/install.sh` | 171 | App install/uninstall picker/orchestrator |
|
||||
|
||||
|
||||
+12
-1
@@ -55,7 +55,7 @@ Category-less tools (`config`, `tree`) live outside any category and are documen
|
||||
|
||||
### ai
|
||||
|
||||
**File:** `bin/pos-ai` (provider-agnostic main tool), `bin/pos-ai-gemini` / `bin/pos-ai-openrouter` (backward-compat forwarders → `pos ai --provider <name>`)
|
||||
**File:** `bin/pos-ai` (provider-agnostic main tool), `bin/pos-ai-gemini` / `bin/pos-ai-openrouter` (backward-compat forwarders → `pos ai --provider <name>`), `bin/pos-ai-hf` (Hugging Face model downloader)
|
||||
**Provider adapters:** `lib/ai-providers/gemini.sh`, `lib/ai-providers/openrouter.sh`
|
||||
**Purpose:** AI assistant with pluggable providers. Six subcommands: `ask` (scriptable, persistent session), `capture` (run a command and save its output for `--last`), `chat` (interactive multi-turn REPL), `models` (list available models), `providers` (list providers and config status), and `sessions` (list/clear sessions). Providers handle API-specific logic; the main tool handles sessions, rendering, machine context, and all shared logic.
|
||||
|
||||
@@ -100,6 +100,17 @@ Model precedence: `--model` flag > `AI_MODEL` env > provider-specific fallback (
|
||||
|
||||
**Messaging bridges:** the Telegram and Matrix listeners forward non-command messages starting with `ai ` (case-insensitive) to `pos ai ask` and reply with the model's answer — see [communication → listener](#communication). The Telegram bridge uses one session per chat (`telegram-<chat id>`), the Matrix bridge one per room (`matrix-<room>`).
|
||||
|
||||
`pos ai hf` — Hugging Face model downloader:
|
||||
|
||||
| Command | Behavior |
|
||||
|---------|----------|
|
||||
| `pos ai hf search <query>` | Search Hugging Face models by query (sorted by downloads); prints model ID, download count |
|
||||
| `pos ai hf download <repo-id> [filename]` | Download a file or entire repo from Hugging Face. Creates `<namespace>-<model-name>/` under `HF_DOWNLOAD_DIR` (default `~/.local/share/linux_post_install/ai/models/`). Options: `--branch <rev>` (specific branch), `--gguf` (only `.gguf` files), `--output <dir>` (override download dir). Progress bars to stderr; summary with path and size to stdout. Writes `.hf-meta` JSON (repo-id, branch, files, timestamp) for `list` and `remove` |
|
||||
| `pos ai hf list` | List all downloaded models with size and date |
|
||||
| `pos ai hf remove <repo-id>` | Remove a downloaded model directory and show freed space |
|
||||
|
||||
Auth: `HF_TOKEN` in `~/.config/linux_post_install/ai.env` (same scope as `pos ai`; edit via `pos config ai`). Even for public repos, a token increases rate limits from 500/5min to 1000/5min. Resume: `curl -C -` resumes interrupted downloads. Rate limit handling: on HTTP 429, sleeps `Retry-After` or 60s, retries once.
|
||||
|
||||
### network
|
||||
|
||||
| Command | File | Purpose | Configuration |
|
||||
|
||||
Executable
+495
@@ -0,0 +1,495 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# POS: ai hf — Download AI models from Hugging Face (search, download, manage)
|
||||
# POS_FLAGS: --branch --gguf --output
|
||||
# POS_DEPS: curl jq
|
||||
# POS_CONFIG: ai | ai.env | HF_TOKEN=secret:Hugging Face API token (https://huggingface.co/settings/tokens) | HF_DOWNLOAD_DIR=:Model download directory (default ~/.local/share/linux_post_install/ai/models)
|
||||
# POS_EXAMPLES: pos ai hf search llama 7b | Search Hugging Face for "llama 7b" models
|
||||
# POS_EXAMPLES: pos ai hf download meta-llama/Llama-3.1-8B-Instruct | Download all files from a repo
|
||||
# POS_EXAMPLES: pos ai hf download meta-llama/Llama-3.1-8B-Instruct --gguf | Download only GGUF quantized files
|
||||
# POS_EXAMPLES: pos ai hf download meta-llama/Llama-3.1-8B-Instruct config.json | Download a single file
|
||||
# POS_EXAMPLES: pos ai hf list | List downloaded models
|
||||
# POS_EXAMPLES: pos ai hf remove meta-llama-Llama-3.1-8B-Instruct | Remove a downloaded model
|
||||
|
||||
source "$(dirname "$0")/../lib/common.sh" 2>/dev/null || source "$(dirname "$0")/common.sh"
|
||||
|
||||
# ── Dependencies (before --help) ───────────────────────────────
|
||||
command -v curl &>/dev/null || err "curl not found (install curl)"
|
||||
command -v jq &>/dev/null || err "jq not found (install jq)"
|
||||
|
||||
# ── Config & defaults ──────────────────────────────────────────
|
||||
CONFIG_FILE="${CONFIG_FILE:-$HOME/.config/linux_post_install/ai.env}"
|
||||
HF_TOKEN="${HF_TOKEN:-}"
|
||||
HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}"
|
||||
|
||||
load_hf_config() {
|
||||
[ -f "$CONFIG_FILE" ] || return 0
|
||||
local k v
|
||||
while IFS='=' read -r k v; do
|
||||
[ -n "$k" ] || continue
|
||||
case "$k" in
|
||||
\#*) continue ;;
|
||||
esac
|
||||
v="${v%\"}"; v="${v#\"}"; v="${v%\'}"; v="${v#\'}"
|
||||
v="${v//$'\r'/}"
|
||||
if [ -z "${!k:-}" ]; then
|
||||
export "$k"="$v"
|
||||
fi
|
||||
done < <(grep -E '^[A-Z_]+=' "$CONFIG_FILE" || true)
|
||||
}
|
||||
|
||||
load_hf_config
|
||||
|
||||
# ── Usage ──────────────────────────────────────────────────────
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: pos ai hf <subcommand> [args]
|
||||
|
||||
Hugging Face model downloader — search, download, and manage AI models.
|
||||
|
||||
Subcommands:
|
||||
search <query> Search Hugging Face models
|
||||
download <repo-id> [filename] Download a file or entire repo
|
||||
list List downloaded models
|
||||
remove <repo-id> Remove a downloaded model
|
||||
|
||||
Download options:
|
||||
--branch <rev> Download from a specific branch/revision
|
||||
--gguf Download only .gguf files (inference-ready)
|
||||
--output <dir> Override download directory
|
||||
|
||||
Examples:
|
||||
pos ai hf search llama 7b
|
||||
pos ai hf download meta-llama/Llama-3.1-8B-Instruct
|
||||
pos ai hf download meta-llama/Llama-3.1-8B-Instruct --gguf
|
||||
pos ai hf download meta-llama/Llama-3.1-8B-Instruct config.json
|
||||
pos ai hf download meta-llama/Llama-3.1-8B-Instruct --branch main
|
||||
pos ai hf list
|
||||
pos ai hf remove meta-llama-Llama-3.1-8B-Instruct
|
||||
|
||||
Config (~/.config/linux_post_install/ai.env):
|
||||
HF_TOKEN Hugging Face API token (better rate limits for public repos)
|
||||
HF_DOWNLOAD_DIR Model download directory (default ~/.local/share/linux_post_install/ai/models)
|
||||
|
||||
Exit codes:
|
||||
0 success
|
||||
1 error (missing deps, invalid input, API failure)
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ── Parse global flags ─────────────────────────────────────────
|
||||
SUBCMD=""
|
||||
SUBCMD_ARGS=()
|
||||
BRANCH=""
|
||||
GGUF_ONLY=0
|
||||
OUTPUT_DIR=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-h|--help) usage ;;
|
||||
--branch)
|
||||
[ $# -ge 2 ] || err "--branch requires a value"
|
||||
BRANCH="$2"; shift 2 ;;
|
||||
--gguf)
|
||||
GGUF_ONLY=1; shift ;;
|
||||
--output)
|
||||
[ $# -ge 2 ] || err "--output requires a value"
|
||||
OUTPUT_DIR="$2"; shift 2 ;;
|
||||
-*)
|
||||
err "Unknown option '$1' (see --help)" ;;
|
||||
*)
|
||||
if [ -z "$SUBCMD" ]; then
|
||||
SUBCMD="$1"
|
||||
else
|
||||
SUBCMD_ARGS+=("$1")
|
||||
fi
|
||||
shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$SUBCMD" ] || usage
|
||||
|
||||
# Apply output dir override
|
||||
if [ -n "$OUTPUT_DIR" ]; then
|
||||
HF_DOWNLOAD_DIR="$OUTPUT_DIR"
|
||||
fi
|
||||
HF_DOWNLOAD_DIR="${HF_DOWNLOAD_DIR:-$HOME/.local/share/linux_post_install/ai/models}"
|
||||
|
||||
# ── Token warning ──────────────────────────────────────────────
|
||||
if [ -z "$HF_TOKEN" ]; then
|
||||
warn "No HF_TOKEN set — using anonymous access"
|
||||
fi
|
||||
|
||||
# ── HF API helpers ─────────────────────────────────────────────
|
||||
HF_BASE="https://huggingface.co"
|
||||
HF_API_BASE="https://huggingface.co/api"
|
||||
|
||||
hf_auth_header() {
|
||||
if [ -n "$HF_TOKEN" ]; then
|
||||
printf 'Authorization: Bearer %s' "$HF_TOKEN"
|
||||
fi
|
||||
}
|
||||
|
||||
hf_api() {
|
||||
local endpoint="$1"
|
||||
local url="${HF_API_BASE}${endpoint}"
|
||||
local auth_header
|
||||
auth_header="$(hf_auth_header)"
|
||||
|
||||
local http_code body tmpfile
|
||||
tmpfile="$(mktemp)"
|
||||
|
||||
local curl_args=(-sS -w '%{http_code}' -o "$tmpfile" --max-time 30)
|
||||
if [ -n "$auth_header" ]; then
|
||||
curl_args+=(-H "$auth_header")
|
||||
fi
|
||||
|
||||
# Rate limit retry: on 429, sleep and retry once
|
||||
local attempt=0
|
||||
while [ $attempt -lt 2 ]; do
|
||||
http_code="$(curl "${curl_args[@]}" "$url" 2>/dev/null)" || {
|
||||
rm -f "$tmpfile"
|
||||
err "Connection timed out — check network"
|
||||
}
|
||||
|
||||
if [ "$http_code" = "429" ]; then
|
||||
local retry_after
|
||||
retry_after="$(curl -sI -H "${auth_header:-}" "$url" 2>/dev/null | grep -i 'retry-after:' | tr -d '\r' | awk '{print $2}')"
|
||||
retry_after="${retry_after:-60}"
|
||||
warn "Rate limited — waiting ${retry_after}s before retry"
|
||||
sleep "$retry_after"
|
||||
attempt=$((attempt + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
break
|
||||
done
|
||||
|
||||
body="$(cat "$tmpfile")"
|
||||
rm -f "$tmpfile"
|
||||
|
||||
case "$http_code" in
|
||||
200) ;;
|
||||
401|403) err "Authentication failed — check HF_TOKEN (pos config ai)" ;;
|
||||
404) err "Model not found: ${endpoint#/api/models/}" ;;
|
||||
429) err "Rate limit exceeded — try again later" ;;
|
||||
*) err "API request failed (HTTP $http_code)" ;;
|
||||
esac
|
||||
|
||||
# Validate JSON
|
||||
if ! printf '%s' "$body" | jq empty 2>/dev/null; then
|
||||
err "Failed to parse API response — check network or HF status"
|
||||
fi
|
||||
|
||||
printf '%s' "$body"
|
||||
}
|
||||
|
||||
hf_repo_files() {
|
||||
local repo_id="$1"
|
||||
local branch="${2:-main}"
|
||||
local ns="${repo_id%%/*}"
|
||||
local repo="${repo_id#*/}"
|
||||
|
||||
if [ "$ns" = "$repo" ]; then
|
||||
err "Invalid repo format: use namespace/model-name"
|
||||
fi
|
||||
|
||||
# Try /tree/ endpoint first (has file sizes + LFS info)
|
||||
local endpoint="/models/${ns}/${repo}/tree/${branch}"
|
||||
local result
|
||||
if result="$(hf_api "$endpoint" 2>/dev/null)"; then
|
||||
printf '%s' "$result"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Fallback: /api/models/{ns}/{repo} (siblings, no sizes)
|
||||
warn "Tree endpoint unavailable, using repo metadata"
|
||||
local fallback
|
||||
fallback="$(hf_api "/models/${ns}/${repo}")" || err "Failed to fetch repo info for $repo_id"
|
||||
printf '%s' "$fallback" | jq '[.siblings[] | {rfilename: .rfilename, size: (.size // 0)}]'
|
||||
}
|
||||
|
||||
hf_search() {
|
||||
local query="$1"
|
||||
local limit="${2:-10}"
|
||||
local encoded_query
|
||||
encoded_query="$(printf '%s' "$query" | jq -sRr @uri)"
|
||||
local endpoint="/models?search=${encoded_query}&sort=downloads&direction=-1&limit=${limit}"
|
||||
hf_api "$endpoint"
|
||||
}
|
||||
|
||||
# ── Utility functions ──────────────────────────────────────────
|
||||
hf_repo_dir() {
|
||||
local repo_id="$1"
|
||||
printf '%s' "$HF_DOWNLOAD_DIR/${repo_id//\//-}"
|
||||
}
|
||||
|
||||
hf_human_size() {
|
||||
local bytes="$1"
|
||||
if [ "$bytes" -ge 1073741824 ]; then
|
||||
awk "BEGIN { printf \"%.1f GB\", $bytes / 1073741824 }"
|
||||
elif [ "$bytes" -ge 1048576 ]; then
|
||||
awk "BEGIN { printf \"%.1f MB\", $bytes / 1048576 }"
|
||||
elif [ "$bytes" -ge 1024 ]; then
|
||||
awk "BEGIN { printf \"%.1f KB\", $bytes / 1024 }"
|
||||
else
|
||||
printf '%d B' "$bytes"
|
||||
fi
|
||||
}
|
||||
|
||||
hf_resolve_branch() {
|
||||
local repo_id="$1"
|
||||
local branch="${2:-}"
|
||||
if [ -n "$branch" ]; then
|
||||
printf '%s' "$branch"
|
||||
return
|
||||
fi
|
||||
# Try to get default branch from API
|
||||
local ns="${repo_id%%/*}"
|
||||
local repo="${repo_id#*/}"
|
||||
local meta
|
||||
if meta="$(hf_api "/models/${ns}/${repo}" 2>/dev/null)"; then
|
||||
local default_branch
|
||||
default_branch="$(printf '%s' "$meta" | jq -r '.defaultBranch // empty' 2>/dev/null)"
|
||||
if [ -n "$default_branch" ]; then
|
||||
printf '%s' "$default_branch"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
printf 'main'
|
||||
}
|
||||
|
||||
# ── Download helpers ───────────────────────────────────────────
|
||||
hf_download_file() {
|
||||
local url="$1"
|
||||
local target="$2"
|
||||
local auth_header
|
||||
auth_header="$(hf_auth_header)"
|
||||
|
||||
local curl_args=(-L -C - --progress-bar -o "$target")
|
||||
if [ -n "$auth_header" ]; then
|
||||
curl_args+=(-H "$auth_header")
|
||||
fi
|
||||
|
||||
if curl "${curl_args[@]}" "$url" 2>&1; then
|
||||
if [ -s "$target" ]; then
|
||||
return 0
|
||||
else
|
||||
warn "Downloaded file is empty: $target"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
warn "Download interrupted for $(basename "$target") (resume with same command)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Subcommands ────────────────────────────────────────────────
|
||||
|
||||
cmd_search() {
|
||||
local query="${SUBCMD_ARGS[*]:-}"
|
||||
[ -n "$query" ] || err "Usage: pos ai hf search <query>"
|
||||
|
||||
local result
|
||||
result="$(hf_search "$query" "10")"
|
||||
|
||||
local count
|
||||
count="$(printf '%s' "$result" | jq 'length')"
|
||||
[ "$count" -gt 0 ] || { warn "No models found for '$query'"; return 0; }
|
||||
|
||||
printf 'Found %d models for "%s":\n' "$count" "$query"
|
||||
printf '%s' "$result" | jq -r '.[] | " \(.id)\t\(.downloads // 0)\t\(.likes // 0)"' | \
|
||||
while IFS=$'\t' read -r id downloads likes; do
|
||||
local dl_str
|
||||
if [ "$downloads" -ge 1000 ]; then
|
||||
dl_str="$(awk "BEGIN { printf \"%.1fk\", $downloads / 1000 }")"
|
||||
else
|
||||
dl_str="$downloads"
|
||||
fi
|
||||
printf ' %-55s %s downloads\n' "$id" "$dl_str"
|
||||
done
|
||||
}
|
||||
|
||||
cmd_download() {
|
||||
local repo_id="${SUBCMD_ARGS[0]:-}"
|
||||
[ -n "$repo_id" ] || err "Usage: pos ai hf download <repo-id> [filename]"
|
||||
|
||||
# Validate repo-id contains /
|
||||
[[ "$repo_id" == */* ]] || err "Invalid repo format: use namespace/model-name"
|
||||
|
||||
local filename="${SUBCMD_ARGS[1]:-}"
|
||||
local branch
|
||||
branch="$(hf_resolve_branch "$repo_id" "$BRANCH")"
|
||||
|
||||
# Get file list from API
|
||||
local files_json
|
||||
files_json="$(hf_repo_files "$repo_id" "$branch")"
|
||||
|
||||
# Filter files
|
||||
local filtered_files
|
||||
if [ -n "$filename" ]; then
|
||||
# Single file mode
|
||||
filtered_files="$(printf '%s' "$files_json" | jq -c --arg fn "$filename" '[.[] | select(.rfilename == $fn)]')"
|
||||
elif [ "$GGUF_ONLY" -eq 1 ]; then
|
||||
# GGUF filter
|
||||
filtered_files="$(printf '%s' "$files_json" | jq -c '[.[] | select(.rfilename | endswith(".gguf"))]')"
|
||||
else
|
||||
# All files
|
||||
filtered_files="$(printf '%s' "$files_json" | jq -c '.')"
|
||||
fi
|
||||
|
||||
local file_count
|
||||
file_count="$(printf '%s' "$filtered_files" | jq 'length')"
|
||||
[ "$file_count" -gt 0 ] || err "No files to download"
|
||||
|
||||
# Prepare target directory
|
||||
local target_dir
|
||||
target_dir="$(hf_repo_dir "$repo_id")"
|
||||
mkdir -p "$target_dir"
|
||||
|
||||
# Disk space pre-flight check
|
||||
local total_size
|
||||
total_size="$(printf '%s' "$filtered_files" | jq '[.[].size // 0] | add // 0')"
|
||||
if [ "$total_size" -gt 0 ]; then
|
||||
local avail_kb
|
||||
avail_kb="$(df --output=avail "$target_dir" 2>/dev/null | tail -1 | tr -d ' ')"
|
||||
local need_kb=$((total_size / 1024))
|
||||
if [ "$avail_kb" -lt "$need_kb" ]; then
|
||||
local need_human avail_human
|
||||
need_human="$(hf_human_size "$total_size")"
|
||||
avail_human="$(hf_human_size "$((avail_kb * 1024))")"
|
||||
warn "Low disk space: need $need_human, only $avail_human available"
|
||||
fi
|
||||
fi
|
||||
|
||||
local downloaded=0
|
||||
local ns="${repo_id%%/*}"
|
||||
local repo="${repo_id#*/}"
|
||||
|
||||
while IFS= read -r file_json; do
|
||||
local fname fsize
|
||||
fname="$(printf '%s' "$file_json" | jq -r '.rfilename')"
|
||||
fsize="$(printf '%s' "$file_json" | jq -r '.size // 0')"
|
||||
total_size=$((total_size + fsize))
|
||||
|
||||
local url="${HF_BASE}/${ns}/${repo}/resolve/${branch}/${fname}"
|
||||
local target="${target_dir}/${fname}"
|
||||
|
||||
if [ "$file_count" -gt 1 ]; then
|
||||
downloaded=$((downloaded + 1))
|
||||
printf '[%d/%d] Downloading %s...\n' "$downloaded" "$file_count" "$fname" >&2
|
||||
fi
|
||||
|
||||
# Create parent directory
|
||||
mkdir -p "$(dirname "$target")"
|
||||
|
||||
if ! hf_download_file "$url" "$target"; then
|
||||
warn "Failed to download $fname"
|
||||
continue
|
||||
fi
|
||||
done < <(printf '%s' "$filtered_files" | jq -c '.[]')
|
||||
|
||||
# Write metadata
|
||||
local meta_file="${target_dir}/.hf-meta"
|
||||
local timestamp
|
||||
timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
local file_list
|
||||
file_list="$(printf '%s' "$filtered_files" | jq -c '[.[] | .rfilename]')"
|
||||
|
||||
cat > "$meta_file" <<METAEOF
|
||||
{
|
||||
"repo_id": "$repo_id",
|
||||
"branch": "$branch",
|
||||
"downloaded_at": "$timestamp",
|
||||
"files": $file_list
|
||||
}
|
||||
METAEOF
|
||||
|
||||
# Summary
|
||||
if [ "$file_count" -eq 1 ]; then
|
||||
local fname
|
||||
fname="$(printf '%s' "$filtered_files" | jq -r '.[0].rfilename')"
|
||||
local fsize
|
||||
fsize="$(printf '%s' "$filtered_files" | jq -r '.[0].size // 0')"
|
||||
local human_size
|
||||
human_size="$(hf_human_size "$fsize")"
|
||||
printf '📥 Downloaded: %s/%s (%s)\n' "$repo_id" "$fname" "$human_size"
|
||||
printf '📁 %s/%s\n' "$target_dir" "$fname"
|
||||
else
|
||||
local total_human
|
||||
total_human="$(hf_human_size "$total_size")"
|
||||
printf '📥 Downloaded: %s (%d files, %s)\n' "$repo_id" "$file_count" "$total_human"
|
||||
printf '📁 %s/\n' "$target_dir"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_list() {
|
||||
[ -d "$HF_DOWNLOAD_DIR" ] || { warn "No models downloaded yet"; return 0; }
|
||||
|
||||
local models=()
|
||||
while IFS= read -r dir; do
|
||||
[ -d "$dir" ] || continue
|
||||
local meta_file="${dir}/.hf-meta"
|
||||
[ -f "$meta_file" ] || continue
|
||||
models+=("$dir")
|
||||
done < <(find "$HF_DOWNLOAD_DIR" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort)
|
||||
|
||||
[ ${#models[@]} -gt 0 ] || { warn "No models downloaded yet"; return 0; }
|
||||
|
||||
printf 'Downloaded models (%d):\n' "${#models[@]}"
|
||||
for dir in "${models[@]}"; do
|
||||
local meta_file="${dir}/.hf-meta"
|
||||
local name
|
||||
name="$(basename "$dir")"
|
||||
local total_size=0
|
||||
local date_str
|
||||
date_str="$(jq -r '.downloaded_at // "unknown"' "$meta_file" 2>/dev/null | cut -dT -f1)"
|
||||
|
||||
# Calculate total size
|
||||
while IFS= read -r file; do
|
||||
[ -f "$file" ] || continue
|
||||
local size
|
||||
size="$(stat -c%s "$file" 2>/dev/null || echo 0)"
|
||||
total_size=$((total_size + size))
|
||||
done < <(find "$dir" -type f ! -name '.hf-meta' 2>/dev/null)
|
||||
|
||||
local human_size
|
||||
human_size="$(hf_human_size "$total_size")"
|
||||
printf ' %-50s %s %s\n' "$name" "$human_size" "$date_str"
|
||||
done
|
||||
}
|
||||
|
||||
cmd_remove() {
|
||||
local repo_id="${SUBCMD_ARGS[0]:-}"
|
||||
[ -n "$repo_id" ] || err "Usage: pos ai hf remove <repo-id>"
|
||||
|
||||
local target_dir
|
||||
target_dir="$(hf_repo_dir "$repo_id")"
|
||||
|
||||
[ -d "$target_dir" ] || err "Model not found: $repo_id"
|
||||
|
||||
# Calculate size before removal
|
||||
local total_size=0
|
||||
while IFS= read -r file; do
|
||||
[ -f "$file" ] || continue
|
||||
local size
|
||||
size="$(stat -c%s "$file" 2>/dev/null || echo 0)"
|
||||
total_size=$((total_size + size))
|
||||
done < <(find "$target_dir" -type f 2>/dev/null)
|
||||
|
||||
local human_size
|
||||
human_size="$(hf_human_size "$total_size")"
|
||||
|
||||
rm -rf "$target_dir"
|
||||
printf 'Removed: %s (freed %s)\n' "$repo_id" "$human_size"
|
||||
}
|
||||
|
||||
# ── Dispatch ───────────────────────────────────────────────────
|
||||
case "$SUBCMD" in
|
||||
search) cmd_search ;;
|
||||
download) cmd_download ;;
|
||||
list) cmd_list ;;
|
||||
remove) cmd_remove ;;
|
||||
*) err "Unknown subcommand '$SUBCMD' (see --help)" ;;
|
||||
esac
|
||||
@@ -3,6 +3,7 @@
|
||||
# Install: source this file in ~/.bashrc or place in /etc/bash_completion.d/
|
||||
# GEN:START posflags
|
||||
declare -A _pos_flags
|
||||
_pos_flags[ai-hf]="--branch --gguf --output"
|
||||
_pos_flags[communication-matrix-listener]="--enable --disable --status --run"
|
||||
_pos_flags[communication-telegram-listener]="--enable --disable --status --sync-commands --run"
|
||||
_pos_flags[communication-telegram-sender]="--type --caption --parse-mode --no-preview --token --chat-id --markdown"
|
||||
@@ -44,7 +45,7 @@ _pos_subcmds[share-smb-client]="mount unmount list persist unpersist menu"
|
||||
_pos_subcmds[share-smb-server]="status share unshare list adduser deluser reload enable disable menu"
|
||||
_pos_subcmds[system-backup]="menu"
|
||||
_pos_subcmds[system-schedule]="run list config enable disable status migrate menu"
|
||||
_pos_subcmds[ai]="ask chat sessions capture models providers alias gemini openrouter"
|
||||
_pos_subcmds[ai]="ask chat sessions capture models providers alias gemini hf openrouter"
|
||||
# GEN:END possubcmds
|
||||
# GEN:START posconfigscopes
|
||||
declare -a _pos_config_scopes=(ai compose entertainment grab matrix notify scrcpy system telegram ytsync)
|
||||
|
||||
Reference in New Issue
Block a user