Struct Tokens

Source

pub struct Tokens(/* private fields */);

Expand description

Tokens are a thin wrapper over a vector of Token objects

They are usually read from a Mouth and treated as an immutable interface. For access to the inner Token contents, use one of the unlist methods.

Implementations§

Source §

impl Tokens

Source

pub fn new(tokens: Vec<Token>) -> Self

Create a Tokens object from a Vec of individual Token

Source

pub fn unlist(self) -> Vec<Token>

Return a list of the tokens making up this Tokens

Source

pub fn unlist_ref(&self) -> &Vec<Token>

Return a reference to the tokens making up this Tokens

Source

pub fn unlist_mut(&mut self) -> &mut Vec<Token>

Return a mutable reference to the tokens making up this Tokens

Source

pub fn is_empty(&self) -> bool

Are there any tokens at all contained in this Tokens object

Source

pub fn len(&self) -> usize

Number of contained Token entries

Source

pub fn starts_with_text(&self, prefix: &str) -> bool

Zero-alloc equivalent of self.to_string().starts_with(prefix). Walks tokens byte-by-byte into prefix using the same Display semantics as eq_text (COMMENT skipped, ARG prefixed with #). Returns true once the full prefix has been consumed, even if more token text follows.

Source

pub fn eq_text(&self, target: &str) -> bool

Zero-alloc equivalent of self.to_string() == target. Walks the contained tokens byte-by-byte, skipping COMMENT tokens (matching Display for Tokens) and prefixing ARG tokens with # (matching Display for Token). Returns true iff the rendered text exactly equals target. Used by DefMacro bodies that check keyword values like true / false / swapnumber without wanting to allocate a fresh String per invocation.

Source

pub fn revert(self) -> Vec<Token>

Source

pub fn to_number(&self) -> Number

to_number casts back to a parsed Number (usually via gullet::read_number) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_dimension(&self) -> Dimension

to_dimension casts back to a parsed Dimension (usually via gullet::read_dimension) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_glue(&self) -> Glue

to_glue casts back to a parsed Glue (usually via gullet::read_glue) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_mu_glue(&self) -> MuGlue

to_mu_glue casts back to a parsed MuGlue (usually via gullet::read_mu_glue) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_mu_dimension(&self) -> MuDimension

to_mu_dimension casts back to a parsed MuGlue (usually via gullet::read_mu_glue) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_float(&self) -> Float

to_float casts back to a parsed Float (usually via gullet::read_float) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn to_keyvals(&self) -> Result<KeyVals>

to_keyvals casts back to a parsed KeyVals (usually via a KeyVals parameter type) which had to be re-converted to a Tokens for reentering the expansion flow

Source

pub fn equals(&self, other: Tokens) -> bool

Methods for overloaded ops.

Source

pub fn unwrap_or_default(self) -> Tokens

returns self, for compatibility convenience with Option

Source

pub fn unwrap(&self) -> &Tokens

returns self, for compatibility convenience with Option

Source

pub fn stringify(&self) -> String

A string form which is primarily used for error-reporting

Source

pub fn be_digested(self) -> Result<Digested>

digest the current Tokens

Source

pub fn neutralize(self, extraspecials: &[char]) -> Tokens

neutralize each token

Source

pub fn is_balanced(&self) -> bool

Checks if any BEGIN/END code groups are correctly nested and closed

Source

pub fn substitute_parameters(&self, args: &[Option<Cow<'_, Tokens>>]) -> Self

substitutes the parameters (ARG catcode) in a Tokens list for concrete arguments

Source

pub fn untex(self) -> String

Consumes a Tokens to a string containing TeX that created it (or could have). Note that this is not necessarily the original TeX code; expansions or other substitutions may have taken place.

Design decision: The Perl UnTeX inserts %\n line-breaks (TeX comment + newline) when a token string would exceed 78 characters. The Rust port deliberately omits this feature. Line-break insertion is purely cosmetic and makes test expectations fragile — the %\n appears verbatim in tex= attributes of ltx:Math elements, causing test XML files to contain %
 escape sequences that depend on exact token lengths. We instead always produce compact, single-line output. Test .xml files should not contain %
.

Source

pub fn untex_string(self) -> TeXString

untex, typed — the blessed way to get TeX markup out of a Tokens and back into a tokenizing sink.

Prefer this over untex() whenever the string is destined for Tokenize! / mouth::tokenize / mouth::tokenize_internal: it is the only Tokens→TeXString conversion, so the sink’s signature proves the round trip cannot weld a control word (\v S stays \v S, not \vS). untex() itself is unchanged for the callers that want a plain String (a tex= attribute, a log message, a comparison).

Source

pub fn pack_parameters(self) -> Result<Self>

Packs repeated CC_PARAM tokens into CC_ARG tokens for use as a macro body (and other token lists) Also unwraps \noexpand tokens, since that is also needed for macro bodies (but not strictly part of packing parameters)

Source

pub fn strip_braces(self) -> Self

Trims outer braces (if they balance each other). Strips exactly 1 layer of matching outer braces by default. Should this also trim whitespace? or only if there are braces?

Source

pub fn strip_braces_n(self, layers: usize) -> Self

Trims layers outer brace pairs (if they balance each other). Also trims whitespace outer to the removed braces. Follows the Perl Tokens.pm algorithm: first collects all balanced brace pairs, then strips from outside-in, only removing pairs that span the full remaining width.