Skip to main content

Module watchdog

Module watchdog 

Source
Expand description

Main-level wall-clock watchdog that forcibly aborts the process after a deadline. Complements the cooperative stomach::check_timeout polling for native-code hotspots (Marpa, libxml2, libxslt) that don’t return to the digestion loop. Wall-clock watchdog that forcibly aborts the process after a deadline.

The existing stomach::check_timeout() is a cooperative mechanism — it only fires when the digestion loop polls it. That leaves tight native loops (Marpa precompute / parse, libxml2 post-processing, FFI calls into libxslt, …) completely unguarded: a 60-second timeout can easily turn into 10 minutes if control never returns to the digestion loop.

This module provides a main-level Watchdog that spawns a dedicated thread at construction, wakes after the specified number of seconds, and — if the watchdog has not yet been cancelled — prints a message and calls std::process::abort(). That guarantees the process dies within timeout + poll_interval of the configured deadline, regardless of what the main thread is doing.

§Design notes

  • Uses Arc<AtomicBool> for cancellation. Polling every 100 ms keeps the cancellation latency low without burning CPU.
  • Drop on the Watchdog handle cancels the watchdog thread, so RAII usage (let _wd = Watchdog::new(secs)) is sufficient.
  • We use std::process::abort() rather than panic! because panic may unwind or be caught by a surrounding catch_unwind, which would defeat the safety guarantee. abort() delivers SIGABRT and always terminates the process.
  • The existing cooperative stomach::check_timeout() path is retained: on most conversions it fires before the hard abort, giving callers a nice Err(Fatal) with proper error propagation. The watchdog is a safety net for the pathological cases where cooperative polling doesn’t happen.

§Resource limits

Watchdog::with_limits guards both a wall-clock deadline and a resident-memory ceiling — the two defenses any executable that converts arbitrary input needs. It is the shared guard reused by both cortex_worker (in-process, one paper per process) and the latexml_oxide --server LSP (run inside each forked body child, which self-terminates on breach so the parent reaps it via pipe EOF). The exit codes are distinct so a supervising parent can tell them apart: 124 = wall-clock timeout, 137 = memory ceiling.

§Portability

The wall-clock guard is portable (std::thread + Instant).

Portable: total_memory_bytes and available_disk_bytes answer on Linux, macOS and Windows — sysconf(_SC_PHYS_PAGES)/statvfs are POSIX, and Windows uses GlobalMemoryStatusEx/GetDiskFreeSpaceExW. They back the machine-derived default ceiling (default_ceiling_mib) and the spill-headroom check, so those behave identically on every supported OS.

Enforcement, also portable: process_rss_kb — the half that actually checks live usage against the ceiling — answers on all three: Linux samples /proc/self/status, macOS asks libproc (proc_pidinfo), and Windows uses GetProcessMemoryInfo. So the memory ceiling is both computed and checked on every supported OS; other targets fall back to None (time guard only).

Structs§

Watchdog

Constants§

EXIT_OOM
Exit code used when the memory ceiling is exceeded (128 + SIGKILL).
EXIT_TIMEOUT
Exit code used when the wall-clock deadline is exceeded (standard timeout).
FALLBACK_CEILING_MIB
Ceiling used when the machine’s RAM cannot be probed — the historical flat default, kept so an unprobeable platform behaves exactly as before rather than losing its guard entirely.
MAX_DEFAULT_CEILING_MIB
Upper bound on the machine-derived default ceiling (64 GiB in MiB).
MIN_DEFAULT_CEILING_MIB
Floor for the machine-derived default, so a small container still gets a workable budget rather than a ceiling no conversion can fit under.

Functions§

available_disk_bytes
Free space in bytes on the filesystem holding path, or None if it cannot be determined. Used to check headroom before spilling intermediates to disk.
available_memory_bytes
RAM available for new allocations right now, in bytes, or None when the platform offers no honest answer.
default_ceiling_mib
The default per-conversion memory ceiling in MiB, derived from the machine as it is right now.
note_memory_fatal
Record that a memory-budget Fatal fired, so the end-of-run report knows memory was actually the problem.
peak_memory_report
The end-of-run memory report — emitted ONLY when a memory-budget Fatal fired during the run (user directive 2026-08-03: alert when needed, stay quiet on clean runs). None otherwise, or when no peak is measurable.
process_peak_rss_kb
Kernel-tracked PEAK resident set of this process in KiB (VmHWM), or None if it can’t be determined. Unlike process_rss_kb’s point-in-time sample, this is the true high-water mark over the whole process lifetime — no sampling cadence can miss a spike — which makes it the honest basis for the end-of-run “this document needed N” report (peak_memory_report).
process_rss_kb
Current resident set size of this process in KiB, or None if it can’t be determined. Linux reads VmRSS from /proc/self/status; macOS asks libproc (proc_pidinfo/PROC_PIDTASKINFO) — without it the whole --max-memory ceiling silently did not exist on macOS, and the 115_streaming_cli Fatal-contract guard caught exactly that on macOS CI (an over-budget run exited 0). Cheap enough to poll a few times a second.
set_pre_exit_hook
total_memory_bytes
Total physical RAM on this machine in bytes, or None if it cannot be determined.