docs: record telegram listener crash-loop failure and bash errexit lesson
gates / consistency-and-conventions (push) Successful in 25s

- failure_telegram_listener_254.md: /capture->ffmpeg 254 exit chain,
  CHLD-trap blind spot, bare 'wait'-under-set -e daemon kill, systemd
  crash-loop + getUpdates offset reset, flock path divergence, fix
  (set -e-safe reap + persisted offset, fcfa2a5)
- lesson_bash_errexit_wait.md: capturing a failing child's status with
  'wait $pid; rc=$?' under set -euo pipefail aborts the parent;
  always use 'wait $pid 2>/dev/null || rc=$?'
This commit is contained in:
Your Name
2026-09-12 13:53:56 -04:00
parent 59a4c0e1df
commit a6976186d4
2 changed files with 50 additions and 0 deletions
@@ -0,0 +1,27 @@
# Failure: telegram-listener exit-254 crash loop (LattePanda, 2026-09-12)
**Root cause:** No webcam device on host (`/dev/video*` absent). The `/capture` command chain
(`ffmpeg -f v4l2 -i /dev/video0 ...` from telegram_commands.env) fails fast with exit 254.
The listener (bin/pos-communication-telegram-listener) spawns that chain as a bg child; its
SIGCHLD trap `while _p=$(wait -n 2>/dev/null); do _EXIT_CODES[$_p]=$?; done` only stores exits
when `wait -n` returns 0 (non-zero exit → while-condition false → body skipped → pid consumed
but never recorded). `reap_commands()` then falls into `wait "$pid" 2>/dev/null; rc=$?`;
a failing `wait` under `set -euo pipefail` kills the whole shell **before** `rc=$?` runs
(`set -e` does NOT exempt a failing simple command followed by `;`). Verified: raw ffmpeg→254,
exact chain→254, trap_full.sh (byte-identical trap/reap + real ffmpeg child)→EXIT=254.
**Loop persistence:** Restart=always + RestartSec=5 + every instance starts `local offset=0`
and dies before confirming the previous getUpdates → same pending updates re-fetch and
re-execute on every restart → /capture fails 254 again → infinite crash loop (NRestarts 80+
in ~1h). Journal bursts byte-identical across instances (160 execs/16 starts ~10 min).
**Prevention:**
1. reap fallback must tolerate non-zero child exits: `rc=0; wait "$pid" 2>/dev/null || rc=$?`
(or capture inside if/||). Never `wait "$pid"; rc=$?` under set -e.
2. CHLD trap should record non-zero exits: `while _p=$(wait -n 2>/dev/null); do ...; done`
cannot observe a failing exit — use `wait -n; _p=$?`-safe pattern or `|| true` + $? capture.
3. Command map entries should pre-check devices (`[ -e /dev/video0 ] || error`) so a
permanently-broken command cannot crash the daemon.
4. Persist the getUpdates offset (or confirm after processing) so replays stop on restart.
5. Diagnose "slow bot" by checking `systemctl --user show ... -p NRestarts` / journal
`status=254` FIRST — crash-loop downtime beats processing speed by orders of magnitude.