Function resolve_rss_cap 

pub fn resolve_rss_cap() -> Option<u64>

Resolve the effective soft-RSS budget: None = disabled (no ceiling), Some(n) = abort above n bytes. Precedence: the explicit set_memory_cap override, else LATEXML_RSS_CAP_BYTES, else the 4.5 GB default. A cap of 0 from EITHER source resolves to None, so --max-memory=0 / LATEXML_RSS_CAP_BYTES=0 mean “no limit” — not “abort immediately” (a literal 0 compared as rss_bytes > 0 is always true).

Note the env is consulted only when nothing set the override — i.e. only for embedders that never call apply_memory_ceiling. Every path in the latexml_oxide binary calls it, so there --max-memory always wins.