Compare commits

...

1 Commits

Author SHA1 Message Date
Your Name a6976186d4 docs: record telegram listener crash-loop failure and bash errexit lesson
gates / consistency-and-conventions (push) Successful in 25s
- failure_telegram_listener_254.md: /capture->ffmpeg 254 exit chain,
  CHLD-trap blind spot, bare 'wait'-under-set -e daemon kill, systemd
  crash-loop + getUpdates offset reset, flock path divergence, fix
  (set -e-safe reap + persisted offset, fcfa2a5)
- lesson_bash_errexit_wait.md: capturing a failing child's status with
  'wait $pid; rc=$?' under set -euo pipefail aborts the parent;
  always use 'wait $pid 2>/dev/null || rc=$?'
2026-09-12 13:53:56 -04:00
2 changed files with 50 additions and 0 deletions
@@ -0,0 +1,27 @@
# Failure: telegram-listener exit-254 crash loop (LattePanda, 2026-09-12)
**Root cause:** No webcam device on host (`/dev/video*` absent). The `/capture` command chain
(`ffmpeg -f v4l2 -i /dev/video0 ...` from telegram_commands.env) fails fast with exit 254.
The listener (bin/pos-communication-telegram-listener) spawns that chain as a bg child; its
SIGCHLD trap `while _p=$(wait -n 2>/dev/null); do _EXIT_CODES[$_p]=$?; done` only stores exits
when `wait -n` returns 0 (non-zero exit → while-condition false → body skipped → pid consumed
but never recorded). `reap_commands()` then falls into `wait "$pid" 2>/dev/null; rc=$?`;
a failing `wait` under `set -euo pipefail` kills the whole shell **before** `rc=$?` runs
(`set -e` does NOT exempt a failing simple command followed by `;`). Verified: raw ffmpeg→254,
exact chain→254, trap_full.sh (byte-identical trap/reap + real ffmpeg child)→EXIT=254.
**Loop persistence:** Restart=always + RestartSec=5 + every instance starts `local offset=0`
and dies before confirming the previous getUpdates → same pending updates re-fetch and
re-execute on every restart → /capture fails 254 again → infinite crash loop (NRestarts 80+
in ~1h). Journal bursts byte-identical across instances (160 execs/16 starts ~10 min).
**Prevention:**
1. reap fallback must tolerate non-zero child exits: `rc=0; wait "$pid" 2>/dev/null || rc=$?`
(or capture inside if/||). Never `wait "$pid"; rc=$?` under set -e.
2. CHLD trap should record non-zero exits: `while _p=$(wait -n 2>/dev/null); do ...; done`
cannot observe a failing exit — use `wait -n; _p=$?`-safe pattern or `|| true` + $? capture.
3. Command map entries should pre-check devices (`[ -e /dev/video0 ] || error`) so a
permanently-broken command cannot crash the daemon.
4. Persist the getUpdates offset (or confirm after processing) so replays stop on restart.
5. Diagnose "slow bot" by checking `systemctl --user show ... -p NRestarts` / journal
`status=254` FIRST — crash-loop downtime beats processing speed by orders of magnitude.
@@ -0,0 +1,23 @@
# Lesson: `set -e` + `wait "$pid"; rc=$?` kills the shell on non-zero child exit
**Symptom:** a daemon loop crash-looping with status = a background child's exit code
(set -euo pipefail shell).
**Debugging approach that worked:**
1. Before theorizing, reproduce the failing command chain in isolation AND check the exact
exit code of each link (ffmpeg on absent device returned 254 — not the "expected" 231
from /dev/null!).
2. Bisect the propagation path: raw command → chain wrapper → bg child spawn → CHLD trap
reaping → reap_commands wait fallback. Using `bash -x` on a byte-identical reduced
script (trap_full.sh) pinpointed the death at `+ wait <pid>`.
3. Baseline shell semantics test: `bash -c 'set -e; false; echo REACHED'` → exits without
REACHED. Proves `;` does NOT shield a failing simple command from errexit. Many devs
wrongly assume `cmd; rc=$?` is always safe — it is NOT under `set -e`.
4. Bash `wait -n` in a while-condition ALSO conceals non-zero child exits (loop body
skipped) → a "reaping" trap silently loses the failure info → downstream fallbacks
go to the dangerous path.
**Key insight:** with `set -euo pipefail`, ANY background failure can become a main-shell
death if reaped via `wait "$pid"` in a plain command list. The safe capture is
`rc=0; wait "$pid" 2>/dev/null || rc=$?` (errexit suppressed by `||`), or run reaping
inside `if wait ...; then ... else rc=$?; fi`.