Skip to content

Reject Unicode escapes outside literal and comment contents - #246

Merged
abashev merged 1 commit into
21.X.Xfrom
abashev/reject-unicode-escapes-outside-literals-comments
Nov 19, 2026
Merged

abashev merged 1 commit into
21.X.Xfrom
abashev/reject-unicode-escapes-outside-literals-comments

Conversation

@abashev

@abashev abashev commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #244. Ports google/google-java-format#1527; independent of the other open pull requests; based on 21.X.X.

javac translates Unicode escapes before it lexes, but the formatter works on the source text, so the two can disagree about where a token starts and ends. JavaInput.buildToks now rejects these inputs up front, with a diagnostic at the line and column of the escape or character:

  • an escape in whitespace or in a comment delimiter, such as // \u000D, whose escaped carriage return ends the comment early;
  • an escape in an identifier, keyword, operator or numeric literal;
  • an escape in a string, character or text block delimiter, or in a line terminator of a text block;
  • an identifier-ignorable character, such as U+200B, in a name or keyword, which javac drops.

Escapes in the contents of comments, string and character literals and text blocks are still allowed and kept as they are.

What changes for users

On 21.X.X both upstream reports reproduce with an error that points nowhere useful:

<stdin>:1:1: error: expected token: '\'; generated } instead
<stdin>:3:5: error: expected token: 'getAndIncrement​'; generated getAndIncrement instead

With the change:

<stdin>:2:6: error: Unicode escapes not allowed in whitespace or comment delimiters
<stdin>:3:20: error: Ignorable characters not allowed in identifiers or keywords

Formatted output for accepted input does not change. The old "Unicode escapes not allowed in whitespace or multi-character operators" error, which had no position, is gone.

Implementation

  • JavaInput: every tok now holds its source text; the checks and their helpers (checkLiteral, indexOfUnicodeLineTerminatorEscape, indexOfIgnorable, error) follow upstream. continueTrailingComment is unchanged.
  • JavacTokens.RawTok no longer carries javac's string value, which nothing reads now.
  • JavaInput.Tok still takes an original and an unescaped text, as upstream's does; the two are now always the same.

Tests

  • DiagnosticTest.unicodeEscapes and ignorableCharacters: the upstream cases, with exit code 2, ours for an input that cannot be formatted. On 21.X.X both fail with the errors above.
  • FormatterTest.unicodeEscapesInLiteralAndCommentContents: escapes in comment and literal contents format unchanged, in our style. It passes before and after.

Results:

  • JDK 21: ./gradlew check passes.
  • JDK 25: DiagnosticTest and FormatterTest pass.

javac translates Unicode escapes before it lexes, but the formatter
works on the source text, so the two can disagree about where a token
starts and ends (#244):

    class Foo {
      // \u000D
    }

The escaped carriage return ends the comment for javac, and the escape
is left in the gap before `}`, where JavaInput took it for a token:

    <stdin>:1:1: error: expected token: '\'; generated } instead

An escape in a comment or literal delimiter, or in a line terminator of
a text block, does the same, and javac also drops identifier-ignorable
characters such as U+200B from names and keywords, so a zero-width
space at the end of `getAndIncrement` failed with "expected token:
'getAndIncrement<U+200B>'; generated getAndIncrement instead".

JavaInput.buildToks now rejects these inputs up front with a diagnostic
at the line and column of the escape or character, as
google/google-java-format#1527 does:

    <stdin>:2:6: error: Unicode escapes not allowed in whitespace or comment delimiters
    <stdin>:3:20: error: Ignorable characters not allowed in identifiers or keywords

Escapes in the contents of comments, string and character literals and
text blocks are still allowed and kept as they are. Every tok now holds
its source text, so RawTok no longer carries javac's string value, and
the old check for escapes in whitespace and multi-character operators,
which reported no position, is gone.

The cases in DiagnosticTest.unicodeEscapes and ignorableCharacters
come from upstream; their exit code is 2, ours for an input that does
not parse. FormatterTest.unicodeEscapesInLiteralAndCommentContents
checks that escapes in contents format unchanged, in our style.
@abashev
abashev merged commit 34a1fe9 into 21.X.X Nov 19, 2026
16 of 17 checks passed
@abashev
abashev deleted the abashev/reject-unicode-escapes-outside-literals-comments branch November 19, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject Unicode escapes outside literals/comments and ignorable characters, with a clear diagnostic

1 participant