Repository navigation
Reject Unicode escapes outside literal and comment contents - #246
Merged
abashev merged 1 commit intoNov 19, 2026
Merged
Conversation
javac translates Unicode escapes before it lexes, but the formatter works on the source text, so the two can disagree about where a token starts and ends (#244): class Foo { // \u000D } The escaped carriage return ends the comment for javac, and the escape is left in the gap before `}`, where JavaInput took it for a token: <stdin>:1:1: error: expected token: '\'; generated } instead An escape in a comment or literal delimiter, or in a line terminator of a text block, does the same, and javac also drops identifier-ignorable characters such as U+200B from names and keywords, so a zero-width space at the end of `getAndIncrement` failed with "expected token: 'getAndIncrement<U+200B>'; generated getAndIncrement instead". JavaInput.buildToks now rejects these inputs up front with a diagnostic at the line and column of the escape or character, as google/google-java-format#1527 does: <stdin>:2:6: error: Unicode escapes not allowed in whitespace or comment delimiters <stdin>:3:20: error: Ignorable characters not allowed in identifiers or keywords Escapes in the contents of comments, string and character literals and text blocks are still allowed and kept as they are. Every tok now holds its source text, so RawTok no longer carries javac's string value, and the old check for escapes in whitespace and multi-character operators, which reported no position, is gone. The cases in DiagnosticTest.unicodeEscapes and ignorableCharacters come from upstream; their exit code is 2, ours for an input that does not parse. FormatterTest.unicodeEscapesInLiteralAndCommentContents checks that escapes in contents format unchanged, in our style.
abashev
deleted the
abashev/reject-unicode-escapes-outside-literals-comments
branch
November 19, 2026 16:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #244. Ports google/google-java-format#1527; independent of the other open pull requests; based on
21.X.X.javac translates Unicode escapes before it lexes, but the formatter works on the source text, so the two can disagree about where a token starts and ends.
JavaInput.buildToksnow rejects these inputs up front, with a diagnostic at the line and column of the escape or character:// \u000D, whose escaped carriage return ends the comment early;Escapes in the contents of comments, string and character literals and text blocks are still allowed and kept as they are.
What changes for users
On
21.X.Xboth upstream reports reproduce with an error that points nowhere useful:With the change:
Formatted output for accepted input does not change. The old "Unicode escapes not allowed in whitespace or multi-character operators" error, which had no position, is gone.
Implementation
JavaInput: every tok now holds its source text; the checks and their helpers (checkLiteral,indexOfUnicodeLineTerminatorEscape,indexOfIgnorable,error) follow upstream.continueTrailingCommentis unchanged.JavacTokens.RawTokno longer carries javac's string value, which nothing reads now.JavaInput.Tokstill takes an original and an unescaped text, as upstream's does; the two are now always the same.Tests
DiagnosticTest.unicodeEscapesandignorableCharacters: the upstream cases, with exit code 2, ours for an input that cannot be formatted. On21.X.Xboth fail with the errors above.FormatterTest.unicodeEscapesInLiteralAndCommentContents: escapes in comment and literal contents format unchanged, in our style. It passes before and after.Results:
./gradlew checkpasses.DiagnosticTestandFormatterTestpass.