pub fn tokenize(text: impl Into<TeXString>) -> TokensExpand description
Tokenize a string under the standard catcode table — Perl
Package.pm:Tokenize L1019-1023.
“Standard” is the document-level table: @ is an ordinary letter-less
character, so this is how user-facing text should be read. The current
state’s catcodes are deliberately NOT consulted; the table is swapped in for
the duration and restored afterwards, exactly as Perl’s local $STATE = $STD_CATTABLE does, so a document that has been playing with catcodes
cannot change what a binding’s own string means.
See tokenize_internal for the .sty-style table that treats @ as a
letter.
The argument is an impl Into<TeXString>, not a &str: a string
literal converts implicitly, but a String — the shape a control-word-welding
Tokens::to_string() arrives in — must declare itself via
Tokens::untex_string or TeXString::assembled. See the TeXString
docs for why.