From edce62582362159274213f74a7c99b21866a7a52 Mon Sep 17 00:00:00 2001 From: Arturo Bernal Date: Thu, 19 Nov 2026 10:43:35 +0100 Subject: [PATCH] Fix public suffix matching for IDN labels unassigned in IDNA2003 Problem: java.net.IDN enforces the Unicode 3.2 repertoire (IDNA2003) by default. A Public Suffix List label whose script was assigned to Unicode after 3.2 - for example Balinese (Unicode 5.0), the script behind the real PSL entry xn--9tfky.id - is "unassigned" under that repertoire. IDN.toUnicode silently fails to decode such a label instead of throwing, so DnsUtils.normalizeUnicode and PublicSuffixMatcher.getDomainRoot never see the label in the Unicode form the PSL rules are keyed by, and the matcher falls through to treating it as an ordinary registrable label one level short of the real suffix. This is a real, exploitable defect: a cookie set with Domain=xn--9tfky.id by one tenant under that suffix is accepted by another, unrelated tenant under the same suffix, defeating the Public Suffix List's whole purpose on the cookie path. The same decode is shared with DefaultHostnameVerifier (matchDNSName/matchCN normalize through the same DnsUtils.normalizeUnicode before calling PublicSuffixMatcher.verifyInternal), so a non-wildcard TLS identity for the bare suffix itself is wrongly accepted too. Solution: Pass IDN.ALLOW_UNASSIGNED to the IDN.toUnicode/toASCII calls in DnsUtils.normalizeUnicode and PublicSuffixMatcher.getDomainRoot. DnsUtils.normalizeUnicode is the fix that actually matters: it is the normalization verify() and DefaultHostnameVerifier both go through. getDomainRoot has no production caller in this module today, but is public API and should stay consistent with verify() for external callers. A separate, pre-existing defect was found while testing this: a wildcard SAN such as *.xn--9tfky.id (or, identically, a plain ASCII *.id) is accepted by verifyInternal for a host anywhere under that suffix, because the literal '*' is not stripped before the suffix lookup. It reproduces identically with or without IDN involved, so it is unrelated to this fix and intentionally left alone here; resolveDomainRoot is not touched. Tests: - TestPublicSuffixMatcher#testUnassignedCodePoint: getDomainRoot, matches and verify against a crafted PSL rule whose Unicode label is unassigned under 3.2 (xn--9tfky.test / Balinese, added to the suffixlistmatcher.txt fixture), in both ACE and Unicode form, plus the same verify() check against the real, live Mozilla Public Suffix List entry (xn--9tfky.id). - TestDefaultHostnameVerifier#testMatchDNSNameUnassignedCodePoint: matchDNSName, through the real normalize-then-verifyInternal call path (not matchIdentity directly, which does not normalize), rejects a non-wildcard certificate for the bare suffix, matching existing ASCII behavior for a control such as gov.uk. - TestRFC6265CookieSpec#testNoArgSpecsRejectUnassignedCodePointSuffix: the real, no-arg RFC6265LaxSpec/RFC6265StrictSpec (which wire the default PublicSuffixMatcher), same pattern as the existing testNoArgSpecsUsePublicSuffixMatcher - a cookie for the bare suffix does not match a different tenant under it, while a cookie for a genuine registrable domain under the suffix still matches its siblings. All three were run RED against unpatched master (production sources reverted to the upstream master blob, in an otherwise identical worktree) and GREEN with the fix, deterministically (3 repeated runs, no sockets, no timing). Full httpclient5 module: 1169/1169. httpclient5-testing is untouched by this revision (an earlier, larger version of this change added integration-server tests there; they duplicated what the CookieSpec-level test already covers and were removed). Compatibility: Observable change: a non-wildcard TLS identity, or a cookie Domain, exactly equal to a PSL suffix whose label was previously silently left undecoded is no longer treated as an ordinary registrable domain. This is a correction, not a new restriction: it brings these labels in line with how any other multi-label suffix (assigned or ASCII) already behaves. Reporter: Jonas Mandalunes (attribution as given). --- .../client5/http/psl/PublicSuffixMatcher.java | 6 ++- .../hc/client5/http/utils/DnsUtils.java | 6 ++- .../impl/cookie/TestRFC6265CookieSpec.java | 28 ++++++++++++++ .../http/psl/TestPublicSuffixMatcher.java | 34 +++++++++++++++++ .../http/ssl/TestDefaultHostnameVerifier.java | 38 +++++++++++++++++++ .../src/test/resources/suffixlistmatcher.txt | 4 ++ 6 files changed, 113 insertions(+), 3 deletions(-) diff --git a/httpclient5/src/main/java/org/apache/hc/client5/http/psl/PublicSuffixMatcher.java b/httpclient5/src/main/java/org/apache/hc/client5/http/psl/PublicSuffixMatcher.java index 6007a08ba4..a36f52cef1 100644 --- a/httpclient5/src/main/java/org/apache/hc/client5/http/psl/PublicSuffixMatcher.java +++ b/httpclient5/src/main/java/org/apache/hc/client5/http/psl/PublicSuffixMatcher.java @@ -141,12 +141,14 @@ public String getDomainRoot(final String domain, final DomainType expectedType) String normalized = DnsUtils.normalize(domain); final boolean punyCoded = normalized.contains("xn-"); if (punyCoded) { - normalized = IDN.toUnicode(normalized); + // See DnsUtils.normalizeUnicode: some PSL labels are unassigned under the + // Unicode 3.2 repertoire IDN enforces by default and still need to decode. + normalized = IDN.toUnicode(normalized, IDN.ALLOW_UNASSIGNED); } final DomainRootInfo match = resolveDomainRoot(normalized, expectedType); String domainRoot = match != null ? match.root : null; if (domainRoot != null && punyCoded) { - domainRoot = IDN.toASCII(domainRoot); + domainRoot = IDN.toASCII(domainRoot, IDN.ALLOW_UNASSIGNED); } return domainRoot; } diff --git a/httpclient5/src/main/java/org/apache/hc/client5/http/utils/DnsUtils.java b/httpclient5/src/main/java/org/apache/hc/client5/http/utils/DnsUtils.java index 05a08b3cd1..fd90135483 100644 --- a/httpclient5/src/main/java/org/apache/hc/client5/http/utils/DnsUtils.java +++ b/httpclient5/src/main/java/org/apache/hc/client5/http/utils/DnsUtils.java @@ -84,7 +84,11 @@ public static String normalizeUnicode(final String s) { } String decoded; try { - decoded = IDN.toUnicode(s); + // Unassigned code points must still decode: the Public Suffix List contains + // labels (for example Balinese, Unicode 5.0) that are unassigned under the + // Unicode 3.2 repertoire IDN enforces by default, and are stored in the PSL + // in decoded form. + decoded = IDN.toUnicode(s, IDN.ALLOW_UNASSIGNED); } catch (final IllegalArgumentException ignore) { decoded = s; } diff --git a/httpclient5/src/test/java/org/apache/hc/client5/http/impl/cookie/TestRFC6265CookieSpec.java b/httpclient5/src/test/java/org/apache/hc/client5/http/impl/cookie/TestRFC6265CookieSpec.java index 342590ede3..2b5507da99 100644 --- a/httpclient5/src/test/java/org/apache/hc/client5/http/impl/cookie/TestRFC6265CookieSpec.java +++ b/httpclient5/src/test/java/org/apache/hc/client5/http/impl/cookie/TestRFC6265CookieSpec.java @@ -93,6 +93,34 @@ void testNoArgSpecsUsePublicSuffixMatcher() throws Exception { } } + /** + * xn--9tfky.id (ᬩᬮᬶ.id) is a real Mozilla Public Suffix List entry whose + * Unicode label is unassigned under the Unicode 3.2 repertoire java.net.IDN enforces by + * default. Without IDN.ALLOW_UNASSIGNED the matcher never decodes it, so it falls + * through to treating it as an ordinary registrable domain instead of a public suffix. + */ + @Test + void testNoArgSpecsRejectUnassignedCodePointSuffix() throws Exception { + final CookieOrigin evilTenant = new CookieOrigin("evil-tenant.xn--9tfky.id", 80, "/", false); + final CookieOrigin otherTenant = new CookieOrigin("other-tenant.xn--9tfky.id", 80, "/", false); + final CookieOrigin legitOrigin = new CookieOrigin("foo.legit.xn--9tfky.id", 80, "/", false); + final CookieOrigin legitSibling = new CookieOrigin("bar.legit.xn--9tfky.id", 80, "/", false); + final List cookieSpecs = Arrays.asList(new RFC6265LaxSpec(), new RFC6265StrictSpec()); + + for (final CookieSpec cookieSpec : cookieSpecs) { + // A cookie for the bare public suffix itself must not reach another tenant under it. + final Cookie suffixCookie = cookieSpec.parse( + new BasicHeader("Set-Cookie", "name=value; Domain=xn--9tfky.id"), evilTenant).get(0); + Assertions.assertFalse(cookieSpec.match(suffixCookie, otherTenant)); + + // A cookie for a genuine registrable domain under the suffix is still a valid + // cookie domain and reaches siblings under it, like any ordinary domain. + final Cookie legitCookie = cookieSpec.parse( + new BasicHeader("Set-Cookie", "name=value; Domain=legit.xn--9tfky.id"), legitOrigin).get(0); + Assertions.assertTrue(cookieSpec.match(legitCookie, legitSibling)); + } + } + @Test void testPathMatchingWithTrailingSlash() throws Exception { final CookieSpec cookieSpec = new RFC6265StrictSpec(); diff --git a/httpclient5/src/test/java/org/apache/hc/client5/http/psl/TestPublicSuffixMatcher.java b/httpclient5/src/test/java/org/apache/hc/client5/http/psl/TestPublicSuffixMatcher.java index b0dd783881..8af854691b 100644 --- a/httpclient5/src/test/java/org/apache/hc/client5/http/psl/TestPublicSuffixMatcher.java +++ b/httpclient5/src/test/java/org/apache/hc/client5/http/psl/TestPublicSuffixMatcher.java @@ -203,6 +203,40 @@ void testVerifyUnicode() { Assertions.assertTrue(matcher.verify("foo.xn--h-2fa.no")); } + /** + * ᬩᬮᬶ.test (A-label xn--9tfky.test) exercises a public suffix whose + * Unicode label is unassigned under Unicode 3.2, unlike hå.no above whose å + * has been assigned since Unicode 1.0. {@link java.net.IDN#toUnicode(String)} without + * {@link java.net.IDN#ALLOW_UNASSIGNED} silently fails to decode such a label, so the + * matcher must still recognise it as the public suffix it is, in both the ACE and the + * Unicode form, and a genuine subdomain under it must still be a valid cookie domain. + * Confirmed against the crafted fixture and, for verify(), against the real, unmodified + * Mozilla Public Suffix List too (rule ᬩᬮᬶ.id, A-label xn--9tfky.id). + */ + @Test + void testUnassignedCodePoint() { + Assertions.assertEquals("q.xn--9tfky.test", matcher.getDomainRoot("q.xn--9tfky.test")); + Assertions.assertEquals("q.ᬩᬮᬶ.test", matcher.getDomainRoot("q.ᬩᬮᬶ.test")); + Assertions.assertNull(matcher.getDomainRoot("xn--9tfky.test")); + Assertions.assertNull(matcher.getDomainRoot("ᬩᬮᬶ.test")); + + Assertions.assertTrue(matcher.matches(".xn--9tfky.test")); + Assertions.assertTrue(matcher.matches(".ᬩᬮᬶ.test")); + + // The bare public suffix, in either form, must never be a valid cookie domain. + Assertions.assertFalse(matcher.verify("xn--9tfky.test")); + Assertions.assertFalse(matcher.verify("ᬩᬮᬶ.test")); + // A genuine registrable domain under it is still allowed, in either form. + Assertions.assertTrue(matcher.verify("evil-tenant.xn--9tfky.test")); + Assertions.assertTrue(matcher.verify("evil-tenant.ᬩᬮᬶ.test")); + + // Same, against the real, live Mozilla Public Suffix List, not just the fixture. + Assertions.assertFalse(pslMatcher.verify("xn--9tfky.id")); + Assertions.assertFalse(pslMatcher.verify("ᬩᬮᬶ.id")); + Assertions.assertTrue(pslMatcher.verify("evil-tenant.xn--9tfky.id")); + Assertions.assertTrue(pslMatcher.verify("evil-tenant.ᬩᬮᬶ.id")); + } + private void checkPublicSuffix(final String input, final String expected) { Assertions.assertEquals(expected, pslMatcher.getDomainRoot(input)); } diff --git a/httpclient5/src/test/java/org/apache/hc/client5/http/ssl/TestDefaultHostnameVerifier.java b/httpclient5/src/test/java/org/apache/hc/client5/http/ssl/TestDefaultHostnameVerifier.java index 5e126a15dc..05c9a1dd8e 100644 --- a/httpclient5/src/test/java/org/apache/hc/client5/http/ssl/TestDefaultHostnameVerifier.java +++ b/httpclient5/src/test/java/org/apache/hc/client5/http/ssl/TestDefaultHostnameVerifier.java @@ -455,6 +455,44 @@ void testMatchDNSName() throws Exception { publicSuffixMatcher); } + /** + * matchDNSName normalizes the host and the SAN with DnsUtils.normalizeUnicode before + * calling matchIdentity, which in turn calls PublicSuffixMatcher.verifyInternal directly, + * bypassing PublicSuffixMatcher.verify. ᬩᬮᬶ.test (A-label xn--9tfky.test) + * is a public suffix whose Unicode label is unassigned under Unicode 3.2, so this exercises + * the same decode path as the cookie bug, but on the TLS hostname verification path. + */ + @Test + void testMatchDNSNameUnassignedCodePoint() throws Exception { + // A genuine subdomain under the suffix is a valid identity, in ACE or Unicode form. + DefaultHostnameVerifier.matchDNSName( + "evil-tenant.xn--9tfky.test", + Collections.singletonList(SubjectName.DNS("evil-tenant.xn--9tfky.test")), + publicSuffixMatcher); + DefaultHostnameVerifier.matchDNSName( + "evil-tenant.ᬩᬮᬶ.test", + Collections.singletonList(SubjectName.DNS("evil-tenant.ᬩᬮᬶ.test")), + publicSuffixMatcher); + // A non-wildcard certificate for the bare public suffix itself must be rejected, + // in ACE or Unicode form, exactly as it already is for an ASCII multi-label + // suffix such as gov.uk (verified below as a control, same shape, no IDN involved). + Assertions.assertThrows(SSLException.class, () -> + DefaultHostnameVerifier.matchDNSName( + "xn--9tfky.test", + Collections.singletonList(SubjectName.DNS("xn--9tfky.test")), + publicSuffixMatcher)); + Assertions.assertThrows(SSLException.class, () -> + DefaultHostnameVerifier.matchDNSName( + "ᬩᬮᬶ.test", + Collections.singletonList(SubjectName.DNS("ᬩᬮᬶ.test")), + publicSuffixMatcher)); + Assertions.assertThrows(SSLException.class, () -> + DefaultHostnameVerifier.matchDNSName( + "gov.uk", + Collections.singletonList(SubjectName.DNS("gov.uk")), + publicSuffixMatcher)); + } + @Test void testMatchIdentity() { // Test 1: IDN matching punycode diff --git a/httpclient5/src/test/resources/suffixlistmatcher.txt b/httpclient5/src/test/resources/suffixlistmatcher.txt index 3dcbbf299b..698e8afd11 100644 --- a/httpclient5/src/test/resources/suffixlistmatcher.txt +++ b/httpclient5/src/test/resources/suffixlistmatcher.txt @@ -57,4 +57,8 @@ gov.uk no hå.no +// unicode, Balinese script (Unicode 5.0, unassigned under Unicode 3.2 / IDNA2003) +test +ᬩᬮᬶ.test + // ===END ICANN DOMAINS===