Struct Token

Source

pub struct Token {
    pub text: SymStr,
    pub code: Catcode,
}

Expand description

The core immutable syntactic primitive resulting from TeX’s read-in and expansion process We allow the fields to be public, so that we can use builder macros such as

macro_rules! T_SPACE(() => {
    Token { text: arena::pin_static(" "), code: Catcode::SPACE}
  });

Fields§

§text: SymStr

an arena id the character content for this token

§code: Catcode

a TeX catcode

Implementations§

Source §

impl Token

====================================================================== Accessors.

Source

pub fn new<T: AsRef<str>>(text: T, code: Catcode) -> Self

simple Token constructor, wrapping over text and catcode

Source

pub fn cycle_fingerprint(&self) -> u64

A cheap structural fingerprint for the cycle-detection guard (crate::cycle_guard). Matches PartialEq semantics: SPACE tokens fingerprint by catcode alone (their text is irrelevant to equality). NOT a stable hash across processes — for in-run loop detection only.

Source

pub fn get_cs_name(&self) -> SymStr

Get the CS Name of the token. This is the name that definitions will be stored under; It’s the same for various `different’ BEGIN tokens, eg.

Source

pub fn with_cs_name<R, FnR>(&self, caller: FnR) -> R
where FnR: FnOnce(&str) -> R,

Execute a closure using the CS Name of the token. This is the name that definitions will be stored under; It’s the same for various `different’ BEGIN tokens, eg.

Source

pub fn pin_cs_name(&self) -> SymStr

artificial, but avoids the data race

Source

pub fn get_primitive_name(&self) -> Option<&'static str>

Get the fixed name of a primitive catcode, or empty string otherwise

Source

pub fn get_executable_name(&self) -> String

Get the CS name only if the catcode is executable!

Source

pub fn get_executable_primitive_name(&self) -> Option<&'static str>

Intersect executable and primitive

Source

pub fn get_sym(&self) -> SymStr

Use the ticket representing the interned “text” of the token

Source

pub fn with_str<R, FnR>(&self, caller: FnR) -> R
where FnR: FnOnce(&str) -> R,

Use the interned &str “text” of the token use to_string instead for an owned String with simpler

Source

pub fn is_noexpand_family(&self) -> bool

True for any member of the \special_relax no-expand family — the representation of a \noexpand’d (expandable or undefined) CS/active token. Such a token is a CS whose NAME is \special_relax \x01 <shadowed text>, carrying the shadowed token’s identity PER-TOKEN (faithful to TeX’s no_expand_flag, which preserves cur_cs), while the whole family resolves to \special_relax’s \relax meaning (crate::state::lookup_meaning fallback). The bare \special_relax (no suffix) is the no-shadow case (\dont_expand at end-of-input). \x01 is never valid in a CS name or as an active char, so the encoding is unambiguous.

Source

pub fn noexpand_shadowed(&self) -> Option<Token>

Recover the shadowed token from a \special_relax-family token, if it shadows one (i.e. not the bare \special_relax). The shadowed token is a CS (text begins \) or an active char.

Source

pub fn get_charcode(&self) -> u32

Return the character code of character part of the token, or 256 if it is a control sequence

Source

pub fn get_catcode(&self) -> Catcode

Return the catcode of the token.

Source

pub fn is_executable(&self) -> bool

is the current one

Source

pub fn neutralize(self, extraspecials: &[char]) -> Token

neutralize really should only retroactively imitate what Semiverbatim would have done. So, it needs to neutralize those in SPECIALS NOTE that although ‘%’ gets it’s catcode changed in Semiverbatim, I’m pretty sure we do NOT want to neutralize comments (turn them into Catcode::OTHER) here, since if comments do get into the Tokens, that will introduce weird crap into the stream.

Source

pub fn revert(self) -> Token

A Token reverts to itself

Source

pub fn stringify(&self) -> String

A string form which is primarily used for error-reporting

Source

pub fn to_register(&self) -> Option<Rc<Register>>

Source

pub fn to_number(&self) -> Number

Source

pub fn to_dimension(&self) -> Dimension

Source

pub fn to_mu_dimension(&self) -> MuDimension

Source

pub fn to_glue(&self) -> Glue

Source

pub fn to_mu_glue(&self) -> MuGlue

Source

pub fn to_float(&self) -> Float

Source

pub fn be_digested(self) -> Result<Digested>

Source

pub fn defined_as(&self, other: &Token) -> bool

Check whether the current token is defined as other. That is, whether it is equal to other, or \let to other. other is is presumed to be some “constant”, explicit token, such as T_SPACE or T_CS!("\\endcsname").