Expand description
Streaming XML substrate for fragmented (bounded-memory) conversion. Streaming XML substrate — the shared representation layer for fragmented (bounded-memory) conversion.
The eager pipeline holds one whole-document DOM from Build to the final
write, so peak RSS scales with document size (measured ~1.84 GB per MB of
source on the 131 MB witness — docs/performance/STREAMING_CORE_DESIGN_2026-07-29.md).
Fragmented mode bounds peak RSS by fragment size instead: closed subtrees
are serialized to disk (“spilled”) during Build and re-materialized one at a
time for the later phases. This module is the substrate both halves stand
on; it knows nothing about TeX or the schema:
SegmentStore— the on-disk spill area: numbered segment files under a directory beside the destination (same volume, socrate::watchdog::available_disk_bytesheadroom checks measure the right filesystem). Pass 1 writes parseable,_lxfragment-wrapped segments; pass 2 replaces each with its processed, splice-ready output text.FragmentIndex— the document-global facts that must survive a spill as plain strings (never as node handles into a freed DOM — the historical finalize-SIGSEGV class): spilledxml:ids and which segment holds them,label → id, RDFa prefix declarations.FragmentReader— a streaming iterator over a segment (or any XML file): materializes ONE top-level subtree at a time as an owned, mutable [libxml::tree::Document] viaxmlTextReaderExpand, while the rest of the file stays unparsed.
Activation policy, spill-eligibility, and the placeholder-splice assembly
live with their owners (stomach/document/core_interface for the core
half, latexml_post::stream_split for the post half); this module only
moves bytes and facts.
Structs§
- Fragment
Index - The string-only registry of what spilled content still owes the rest of the conversion.
- Fragment
Reader - Iterates the top-level subtrees of an XML file (the children of its root
element), materializing one owned [
Document] at a time. - Segment
Id - Identifies one spilled segment within its
SegmentStore. - Segment
Meta - Per-segment facts recorded at spill time and consumed when the segment is re-materialized in pass 2.
- Segment
Store - The on-disk spill area: numbered segment files plus their in-RAM metadata.