Report invalid UTF-8 byte in a comment as a parsing error

[ksss]

Follow-up to #2973.

Background

In #2973 @soutaro noted:

It would be better if we can report an error when invalid token is included in the comment, but I cannot figure out how to do that right now...

#2973 stopped the lexer from hanging by advancing one byte past an invalid UTF-8 byte, but a byte that fell inside a comment was then silently swallowed — comments scan until newline/EOF, so # \xC2 parsed successfully.

Approach

The lexer is code-point based (YYCTYPE = unsigned int, fed by rbs_peek). The trick is to give an invalid byte a sentinel code point and exclude it from the comment rule's character class:

• src/lexstate.c: an invalid byte (char_width == 0) now maps to U+FFFD instead of its raw value.
• src/lexer.re: the comment rule becomes "#" (. \ [\x00�])*, so the comment stops at the invalid byte. The catch-all rule (*) then turns it into an ErrorToken, which the parser reports as a normal RBS::ParsingError.

U+FFFD is safe as a sentinel because valid input never produces it as a code point: valid multibyte characters map to the existing dummy code point (12523) and valid single bytes map to their own value (0..255). Even a genuine U+FFFD in the input is a 3-byte sequence that decodes to the dummy code point, not to 0xFFFD.

buf = RBS::Buffer.new(content: "# \xC2".force_encoding("UTF-8"), name: "x.rbs")
RBS::Parser._parse_signature(buf, 0, buf.content.bytesize)
# before: parses successfully (invalid byte swallowed by the comment)
# after:  raises RBS::ParsingError

src/lexer.c is regenerated with re2c 4.3 (matching CI). rake test: 958 tests, 0 failures.

Question on scope

This PR only touches the comment rule, which is the case @soutaro mentioned. The same "scan until a delimiter" shape also appears in string literals (dqstring/sqstring), annotations (%a{...} etc.), quoted identifiers (`...`), and inline comments (--). An invalid byte there is still swallowed today.

Should I extend the same sentinel exclusion to those rules in this PR, or keep this one focused on comments and handle the rest separately? Happy to go either way.

---

This PR description was written by Claude Code.