Skip to main content

Module stream_split

latexml_post

Module stream_split 

Source
Expand description

Streaming split: partition a huge core XML file into per-page spill files without ever building the whole-document DOM.

The DOM Split processor (crate::split::Split, port of Perl Post::Split) needs the entire document parsed first — measured at ~16 GB for a 614 MB core XML, and past what a 32 GB host can parse at all for the 131 MB book witness’s 2.68 GB core XML (OOM before Split, zero pages written; laptop UAT 2026-07-31). This module is the STREAMING_POST_DESIGN_2026-07-06.md §3 front-end: a [libxml::reader::TextReader] pull-parse over the file, materializing one non-page subtree at a time, assembling each page’s XML as text and spilling it the moment the page closes. Peak memory is the open ancestor chain plus one content subtree.

§Fidelity contract

The spill files must re-parse into the same per-page DOMs the whole-DOM pipeline (parse → Split::process → per-page spill) produces — the parity gate is byte-equality of the final rendered pages across the two paths (guard: latexml_oxide/tests/118_streaming_split_parity.rs). Every quirk of Split::process_pages is replicated deliberately:

  • Run adjacency. A TOC (<ltx:TOC><ltx:toclist class="ltx_toclist_X">) is emitted per maximal run of adjacent page siblings; ANY intervening sibling node — whitespace text included — breaks a run, exactly as the entries[0].node == removed[0] check does. (ltx:navigation siblings do NOT break runs: the DOM path excises them before any page surgery.)
  • TOC suppression probe. A run’s TOC is suppressed iff an ltx:TOC[@lists='toc'] (exact match — generated TOCs carry no lists) already occurs among the parent’s descendants at flush time, i.e. in content preceding the run.
  • inlist="toc" propagation is per tree level (all page children of one page, across different DOM parents), with the DOM path’s substring semantics (inlist.contains("toc")). It needs lookahead, so the attribute is patched into already-written spill files afterwards (Splitter::patch_inlist_toc).
  • new_document template copies: every page gets the <?latexml …?> PIs, the document’s ltx:resource elements, the root’s direct ltx:date children (only when the page has no direct ltx:date), the root class merged into its own — in exactly that order — then the saved ltx:navigation elements.
  • Inherited attributes (xml:lang, backgroundcolor): nearest ancestor-or-self value, copied onto each page root.
  • Naming ports Split::{presort,prename,get_page_name} including the FOO{n} unnamed-page counter and its name-level-then-descend ordering.
  • The root page gets xml:id="TEMPORARY_DOCUMENT_ID" when it has no id (the Writer removes it later), and pre-order spill order is the DOM path’s docs order (root first, each page before its descendants).

§Wrapper descent

A non-page subtree that contains page matches (e.g. a back-matter wrapper holding ltx:appendix pages) or ltx:navigation elements cannot be bulk-copied: it is expanded to an owned mini-document ([TextReader::expand_to_document]) and the same split surgery runs on that DOM — page matches are pre-collected on the intact subtree (the DOM path evaluates its predicates before any surgery), nested pages are unlinked and written, TOC elements are inserted in place, and the remaining shell is serialized into the enclosing page. Detection is a conservative substring probe (Splitter::needs_dom_descent) — over-triggering costs one expand+copy, never correctness.

§Out of scope (fail loud, not wrong)

  • Documents whose ltx namespace mapping cannot be resolved from the declaration stack error out with a pointer to LATEXML_POST_STREAM_SPLIT=0 (the whole-DOM path).
  • ltx:resource elements / <?latexml?> PIs first appearing after a page has been written cannot retroactively reach it; a Warn flags the (never observed in practice) case.

Structs§

StreamSplitOutcome
Result of a successful streaming split.
StreamedSplitPage
One spilled page, in pre-order (index 0 is the root page).

Functions§

stream_split
Split a core-XML file into per-page spill files, streaming.
supports_union
Can this --splitpath union be evaluated by the streaming split? (The make_splitpaths grammar parses; a custom hand-written XPath may not.) The driver gates on this so a genuine mid-stream failure can be told apart from “not applicable”.