Skip to main content

decode_input_bytes

Function decode_input_bytes 

Source
pub fn decode_input_bytes(raw: &[u8]) -> String
Expand description

Decode raw input bytes as text when no encoding has been declared: valid UTF-8 is taken as-is, anything else is a Latin-1 passthrough (byte → char).

Mirrors Perl Mouth.pm L75-80: when PERL_INPUT_ENCODING is undef Perl never decodes, so the bytes pass through untouched and the read cannot fail. The point is that a non-UTF-8 file is never lost — only decoded conservatively. std::fs::read_to_string gives the opposite behaviour (hard error on the first stray byte), which silently cost witness 2605.00490 its entire bibliography: a JabRef-written .bib self-declaring % Encoding: Cp1252. Real bibtex 0.99d is 8-bit clean and reads it fine.

Latin-1 (rather than from_utf8_lossy) is the better fallback here because it is lossless byte → char: legacy .bib files are overwhelmingly Latin-1/Cp1252, whose accented names survive intact instead of collapsing to U+FFFD.

The fallback is applied per line, not per buffer. raw is a single line when the Mouth calls this, but a whole file when a .bib reader does, and decoding a whole file as Latin-1 because of one stray byte would mojibake every correctly-UTF-8-encoded name in it (éé). Per-line keeps the damage to the offending line and matches the Mouth’s own granularity. The all-valid-UTF-8 case (the overwhelming majority) still costs exactly one from_utf8 SIMD validation of the whole buffer.