Module gullet 

The Gullet is responsible for reading Tokens and other data from the Mouth

Structs

Gullet

MouthRuntime

Enums

BalancedBoundary

What a balanced read (read_balanced) does when it exhausts a mouth before the braces balance.

ExpansionLevel

A directive describing to what degree a gullet reader should perform TeX’s expansion

Constants

CYCLE_GUARD_ACTIVATE_GRAPHICS

Cycle-guard activation floor for graphics-heavy bindings (pgf/tikz/xy). These packages legitimately expand 100M+ tokens; this floor sits above the heaviest measured healthy graphics doc (1805.03265 tikz-cd ~155M) so they stay out of the per-token fingerprint regime, while remaining far below the 400M token_limit backstop. Raised — never lowered — per raise_cycle_guard_activate.

Statics

GULLET

TRACE_GROUP_END

Cached snapshot of LXML_TRACE_GROUP_END env var, sampled exactly once per process. Inlining std::env::var(...) on the hot read_x_token path was triggering SIGSEGVs in __GI_getenv under concurrent test-thread execution: glibc’s getenv walks the process-global environ array unprotected, and the volume of concurrent calls (millions/sec across N threads) made the unsafe concurrent walks visible. The fix is to read the env var ONCE at static-init time; subsequent checks are a free atomic load.

Functions

close_mouth

do_expand

Fully expand tokens and return the result, without digesting them.

do_expand_partially

As do_expand, but stopping at the first unexpandable token of each branch (ExpansionLevel::Partial) instead of expanding to the bitter end.

final_progress

The conversion’s final token-read progress (for end-of-run telemetry — the calibration basis for token_limit / CYCLE_GUARD_ACTIVATE).

flush

Obscure, but the only way I can think of to End!! (see \bye or \end{document}) Flush all sources (close all pending mouth’s)

flush_mouth

This flushes a mouth so that it will be automatically closed, next time it’s read Corresponds to TeX’s \endinput

get_location

Comment-oriented location string, based on get_locator

get_locator

Get the current location of input getting read

get_locator_from_start

get_locator’s accurate-start sibling (§1, docs/performance/SOURCE_PROVENANCE.md): same mouthstack walk, but reads the mouth’s get_locator_from_start (from = the last token’s captured start) instead of the heuristic from. Used for the construct-START snapshot at constructor digest under --source-map.

get_pending_comment

has_more_input

Check if there is more input to be read from the current mouth

if_next

initialize_gullet

is_column_end

mouth_is_open

open_mouth

open_mouth_with

open_mouth, choosing how a balanced read treats the new mouth’s end.

peek_token

Perl: peekToken — peek at the next token without triggering alignment Sets ALIGN_STATE to 1000000 to suppress alignment template handling (Perl line 331-337)

queue_sizes

Queue sizes (mouthstack, pending_comments) — pass-1 streaming telemetry.

raise_cycle_guard_activate

Raise the cycle-guard activation floor for the current (thread-local) gullet, only ever upward. Called from graphics package bindings (pgf/tikz/xy) whose healthy expansion runs to 100M+ tokens — see CYCLE_GUARD_ACTIVATE_GRAPHICS. Idempotent and order-independent: loading several graphics packages just re-asserts the same floor. The per-conversion reset (initialize_gullet) restores the default so the raise does not leak across documents.

read_arg

Read and return a “normal” TeX argument

read_balanced

Approximates TeX’s scan_toks (but doesn’t parse \def parameter lists) and only optionally requires the openning “{”.

read_cs_name

read_cs_name_quiet

Quiet version of read_cs_name — used by \ifcsname. In TeX, \ifcsname silently skips non-expandable CS tokens and returns the constructed name without emitting errors (unlike \csname which DOES emit errors).

read_dimension

read_factor

Return a number (Rust f64 number)

read_float

====================================================================== Float, a floating point number. Similar to factor, but does NOT accept comma! This is NOT part of TeX, but is convenient.

read_glue

read_keyword

Match the input against a set of keywords; Similar to readMatch, but the keywords are strings, and Case and catcodes are ignored; additionally, leading spaces are skipped. AND, macros are expanded.

read_match

Match the input against one of the Token or Tokens in @choices; return the matching one or undef.

read_mu_dimension

read_mu_glue

read_mu_unit

read_next_conditional

reads and discards tokens, until it encounters a conditional, if any. Perl: skipConditionalBody inner loop (Conditional.pm L127-133) reads tokens directly from pushback/mouth (NOT through readToken) and manually tracks $LaTeXML::ALIGN_STATE for { and }. Critically, this bypasses the “alignment-template trigger” check that fires handleTemplate on &/\cr when align_group_count==0 — that check belongs to digestion, not to \else-skip. Rust’s read_token includes the trigger; calling it from here would let pmatrix’s & get treated as the OUTER alignment’s column-end during \ifx.#1.\else…\fi skip when #1 contains \begin{pmatrix}…&…\end{pmatrix}. (REG-2 / math-ph0501074: the 9-line \nonumber+\lefteqn+\pmatrix repro.) Use read_internal_token instead and track BEGIN/END manually, matching Perl byte-for-byte.

read_non_space

Read a single non-space token

read_normal_integer

read_number

read_optional

Read and return a LaTeX optional argument

read_optional_angled

The angle-bracket twin of read_optional.

read_optional_signs

read_raw_line

Read the next raw line (string); primarily to read from the Mouth, but keep any unread input!

read_register_value

read_register_value_coerce

Read a register value, optionally coercing from a compatible larger type. Perl: readRegisterValue($self, $type, $sign, $coerce) Coercion rules (from Perl %RegisterCoercionTypes): Number <- Dimension, Glue (extract raw i64) Dimension <- Glue (extract skip as Dimension) MuDimension <- MuGlue (extract skip as MuDimension)

read_rubber

read_token

read_token_required

Read a token that the calling macro/primitive REQUIRES, holding the “argument expected but input ended” diagnostic in one place.

read_tokens_value

Apparent behaviour of a token value (ie \toks#=<arg>)

read_unit

Read a unit, returning the exact TeX (num, den) conversion fraction (see convert_unit_ratio / numeric_ops::fixpoint_unit). Internal/coerced units (\wd0, \dimen, glue) yield (value_sp, 65536) — the floor(fix·v/65536) path of tex.web §8983, exact in integer arithmetic.

read_until

Return a (balanced) sequence tokens until a match against one of the Tokens in @delims.

read_until_brace

reads until it encounters a Catcode::BEGIN token Note: Perl uses $$token[1] == CC_BEGIN (catcode check, not defined_as)

read_until_token

Convenience method wrapping around read_until TODO: This seems to be the wrong Rust type interface, we need to rework…

read_value

read_x_non_space

Read a single expanded, non-space, token

read_x_token

Read the next non-expandable token, expanding until one appears. Hot path — read_token is folded in. toplevel (default true): on mouth exhaustion, step to the containing mouth. fully_expand (default = toplevel): expand even protected defns (“for execution”). Unlike read_balanced, does NOT defer \the & friends; \noexpand’d tokens act like \relax. For \if/ \ifx arguments pass for_conditional=true (handles \noexpand and CS \let to tokens specially).

reading_from_mouth

Do something, while reading tokens from a specific Mouth.

restore_token_limit

Restore the token limit and progress from a previous set_token_limit call.

scale_token_limit_to_source

Scale the runaway-token backstop to the SOURCE size. The 400M baseline is sized for arXiv-scale inputs; a book-scale source legitimately expands past it (witness: a 131 MB flat-index compilation died at 400M mid-digestion with no loop in sight, at a measured ~3+ tokens/byte and paper-class documents measured up to ~160 tokens/byte). ×200/byte keeps the ceiling finite — a true infinite loop still trips — while never LOWERING the baseline for small documents. An explicit LATEXML_TOKEN_LIMIT wins unchanged (including 0 = disabled).

set_internal_dimension_fn

Install the internal-dimension resolver (calc.sty at load time).

set_pushback_limit

Set the pushback limit (maximum pushback stack size before fatal error).

set_token_limit

Set the token limit and reset progress. Returns previous (limit, progress) for restoration.

show_unexpected

Perl: showUnexpected — returns a debug message about the next available token

skip_filler

<filler> = <optional spaces> | <filler>\relax<optional spaces>

skip_one_space

Skip one optional space. If expanded is true, acts like <one optional space> and expands tokens (readXToken). Perl: skip1Space($self, $expanded)

skip_spaces

Discard any run of spaces at the head of the input — Perl Package.pm:SkipSpaces.

unread

Push the tokens back into the input stream to be re-read.

unread_mut

Variant of unread, but drains the contents of tokens without taking ownership.

unread_one

Unreads a single Token to the start of the token stream. Perl: unread() always adjusts $ALIGN_STATE when unreading { or } tokens.

unread_vec

Unreads a Vec<Token> to the start of the token stream Perl: also adjusts ALIGN_STATE by retracting scanned braces (Gullet.pm lines 343-358)

with_mouth_mut

Execute a function with a mutable reference to the current mouth

Type Aliases

InternalDimensionFn

A package-installed resolver for control sequences that yield an internal dimension by reading their own argument — e.g. calc.sty’s \widthof{box}. Given the already-read CS token, it either consumes that CS’s argument(s) from the gullet and returns the measured value (Ok(Some(_))), or leaves the stream untouched and declines (Ok(None)), in which case the caller un-reads the token. Keeps core calc-agnostic: the base dimension reader consults this seam before giving up with “Missing number”, so \makebox[\widthof{X}] / \rule{\widthof{X}}{…} resolve like real LaTeX without \widthof having to be a register (which would change its bare digestion). See calc_sty.rs / OXIDIZED_DESIGN #115.