pub fn tokenize_bib_literal(text: impl Into<TeXString>) -> TokensExpand description
Tokenize a string under the standard catcode table, reading % & # as
ordinary characters rather than comment, alignment tab and parameter.
For text that came out of the BibTeX lexer, which has none of those
constructs, so all three are data — treatment 1 of OXIDIZED_DESIGN #74,
see Mouth::with_bib_data_literals (including why _ is not in the set).
Plain tokenize would let a % comment out the rest of the string, which
for a .bib field means losing its closing brace and leaving whatever it
opened unclosed, and would make the & in “Taylor & Francis” a stray
alignment tab.
This exists because the handlers that re-read a raw field — \bib@@title
recasing, name splitting, date/pages assembly — build their tokens from the
stored string and never pass through the per-entry mouth.
Takes an impl Into<TeXString> for the same reason as tokenize —
and it is the sink that most needs it: the bibliography is where the
control-word weld has surfaced three times (PR #399, PR #400, issue 410).