Skip to content

Handle GB18030 length in multibyte case folding - #75

Merged
headius merged 2 commits into
jruby:masterfrom
headius:codex/gb18030-casefold-length-2026
Nov 20, 2026
Merged

headius merged 2 commits into
jruby:masterfrom
headius:codex/gb18030-casefold-length-2026

Conversation

@headius

@headius headius commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

JRuby PR #10121 resolves JCodings 1.0.66, where the shared multibyte case-fold helper calls length(byte). GB18030 passes a null EncLen table because its character length comes from its transition table, so MRI TestGB18030#test_mbc_case_fold fails with an NPE surfaced as RegexpError.

For an encoding without a per-byte length table, the helper now derives the length with the encoding-specific length(bytes, p, bytes.length) implementation. It keeps malformed or truncated lengths positive and clips copying to the actual backing-array boundary.

The source audit found no other helper caller with this root cause:

  • GBK, Big5, Big5-UAO, Big5-HKSCS, EUC-KR, CP949, EUC-TW, and Emacs-Mule provide per-byte EncLen tables.
  • UTF-16/32 use a null table too, but UnicodeEncoding.mbcCaseFold overrides this helper.

The regression tests cover GB18030's 2-byte and 4-byte sequences with the logical end inside the character, plus representative split-boundary cases for every other concrete encoding that calls the shared helper.

Verification on Java 21:

  • JCodings focused tests: TestGB18030, TestEmacsMule, and TestCaseMap — 12 tests, 0 failures/errors.
  • JRuby integration with JCodings 1.0.67-SNAPSHOT passed the bounded GB18030 case-insensitive backreference from test_mri/ruby/enc/test_gb18030.rb.

JRuby PR #10121 needs a released JCodings version containing this fix before its MRI CI matrix can pass.

@headius headius added this to the 1.0.67 milestone Nov 19, 2026
@headius

headius commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

This regression was not found until the released jcodings 1.0.66 hit JRuby's CI. I'm having the agent verify that these changes fix and do not regress any tests in JRuby before proceeding with a second release.

@headius headius left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@headius
headius marked this pull request as ready for review November 20, 2026 00:02
@headius
headius merged commit 30bf7f2 into jruby:master Nov 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant