Module token 

The Token and its Catcode

Structs

Token

The core immutable syntactic primitive resulting from TeX’s read-in and expansion process We allow the fields to be public, so that we can use builder macros such as

Enums

Catcode

A Token category code, as in TeX

Constants

NOEXPAND_PREFIX

Name of the bare no-op control sequence \noexpand collapses to, and the prefix of every member of the no-expand family. See Token::is_noexpand_family.

NOEXPAND_SEP

Separator byte between the \special_relax prefix and the shadowed token’s text in a family token’s name. \x01 is never valid in a CS name or active char, so \special_relax\x01\foo is unambiguous and cannot collide with a real CS such as a (hypothetical) \special_relaxfoo.

Statics

TOKEN_ALIGN

constant for an ALIGN “&” token

TOKEN_BEGIN

constant for a BEGIN “{” token

TOKEN_CR

constant for a CR “\n” token

TOKEN_END

constant for an END “}” token

TOKEN_ENDCSNAME

constant for T_CS(“\endcsname”)

TOKEN_EXPANDAFTER

constant for T_CS(“\expandafter”)

TOKEN_MATH

constant for a MATH “$” token

TOKEN_PARAM

constant for a PARAM “#” token

TOKEN_RELAX

constant for T_CS(“\relax”)

TOKEN_SPACE

constant for a SPACE “ “ token

TOKEN_SUB

constant for a SUB “_” token

TOKEN_SUPER

constant for a SUPER “^” token

Functions

noexpand_family

Build the no-expand family token shadowing shadowed — the per-token, dump-safe representation of \noexpand <shadowed> (faithful to TeX’s no_expand_flag: the shadowed identity is preserved in the name; the family resolves to \relax meaning). shadowed is an expandable/undefined CS or active token (the only things \noexpand wraps; see is_dont_expandable).