Skip to content

Comments: Encode emoji for utf8mb3 comment columns - #12673

Open
bazza wants to merge 19 commits into
WordPress:trunkfrom
bazza:65700-encode-emoji-comments
Open

bazza wants to merge 19 commits into
WordPress:trunkfrom
bazza:65700-encode-emoji-comments

Conversation

@bazza

@bazza bazza commented Jul 24, 2026 •

Copy link
Copy Markdown

On utf8 and utf8mb3 columns, comment writes can fail or lose four-byte UTF-8 characters. Preserve those characters in comment authors and content by storing them as HTML character references. Duplicate detection, previously approved author checks, and comment_exists(), and length limit checks use the same encoding as writes.

Trac ticket: https://core.trac.wordpress.org/ticket/65700

See also: https://core.trac.wordpress.org/ticket/46947

Use of AI Tools

AI assistance: Yes
Tool(s): Claude, Codex
Model(s): Sonnet, Fable, GPT…
Used for: Code abstraction and tests. Human reviewed and tested.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

…nt() and wp_update_comment().

wp_insert_post() already encodes emoji as HTML entities before saving to a
utf8/utf8mb3 column, via wp_encode_emoji(). wp_insert_comment() and
wp_update_comment() had no equivalent handling.

Extracts the shared logic into a new private helper,
_wp_encode_emoji_in_fields( $data, $table, $fields ), in
wp-includes/formatting.php, used by wp_insert_post() (refactor, no
behavior change) and now also by wp_insert_comment() and
wp_update_comment() for the comment_author and comment_content fields.

Includes PHPUnit coverage for the new helper directly, plus
deterministic (pre_get_col_charset-filter-forced) tests for
wp_insert_post(), wp_insert_comment(), and wp_update_comment(), so the
utf8mb3 code path is exercised in CI regardless of the test database's
actual charset.

Props barry.
Fixes #65700.
Copilot AI review requested due to automatic review settings July 24, 2026 02:59
@github-actions

github-actions Bot commented Jul 24, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @barry@a8c.com.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

Core Committers: Use this line as a base for the props when committing in SVN:

Props barry, jonsurrell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub or WordPress.org user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

@github-actions

Copy link
Copy Markdown

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR ensures emoji are safely stored when writing to utf8/utf8mb3 comment columns by introducing a shared helper to HTML-entity encode emoji for legacy 3-byte UTF-8 DB columns, aligning comment insertion/update behavior with the existing post insertion logic.

Changes:

  • Extracts shared emoji-encoding logic into _wp_encode_emoji_in_fields() (private helper) and refactors wp_insert_post() to use it (intended no behavior change).
  • Applies the helper to wp_insert_comment() and wp_update_comment() for comment_author and comment_content.
  • Adds PHPUnit coverage for the helper and deterministic tests that force the utf8mb3 code path via pre_get_col_charset.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/phpunit/tests/post.php Adds a deterministic utf8mb3-path test for wp_insert_post() via pre_get_col_charset.
tests/phpunit/tests/formatting/emoji.php Adds direct unit tests for _wp_encode_emoji_in_fields() behavior and charset alias handling.
tests/phpunit/tests/comment.php Adds deterministic tests for utf8mb3-path encoding in wp_insert_comment() / wp_update_comment(), plus a utf8mb4 control.
src/wp-includes/post.php Refactors emoji encoding in wp_insert_post() to use the new shared helper.
src/wp-includes/formatting.php Introduces _wp_encode_emoji_in_fields() helper used across post/comment insert/update flows.
src/wp-includes/comment.php Applies emoji encoding helper to comment insert/update for legacy utf8/utf8mb3 columns.
Comments suppressed due to low confidence (1)

tests/phpunit/tests/formatting/emoji.php:458

  • This filter callback is invoked via pre_get_col_charset and will be called with arguments; keeping a 0-arg signature can emit warnings. Update the signature to accept the standard $charset, $table, $column parameters (even if unused) so it’s compatible with the hook invocation.
	public function filter_pre_get_col_charset_utf8_alias_for_encode_emoji_in_fields_test() {
		return 'utf8';

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/wp-includes/formatting.php Resolved
Comment thread src/wp-includes/formatting.php Resolved
Comment thread tests/phpunit/tests/comment.php Resolved
… hitting a real DB query.

The test passed a fake 'some_table' table name to _wp_encode_emoji_in_fields()
without short-circuiting wpdb::get_col_charset() via the pre_get_col_charset
filter, unlike the other tests in this file. This caused a real
'SHOW FULL COLUMNS FROM some_table' query against a table that doesn't exist,
failing with a WordPress database error instead of testing the intended
behavior (that a field missing from $data is left alone).

See #65700.
Copilot AI review requested due to automatic review settings July 24, 2026 03:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

tests/phpunit/tests/formatting/emoji.php:465

  • The pre_get_col_charset filter always passes at least the $charset argument (accepted args defaults to 1 when omitted). These callbacks declare zero parameters, which can trigger “too many arguments” warnings/errors under PHPUnit/PHP and make the test suite fail.
	public function filter_pre_get_col_charset_utf8_alias_for_encode_emoji_in_fields_test() {
		return 'utf8';
	}

tests/phpunit/tests/formatting/emoji.php:469

  • The pre_get_col_charset filter passes the $charset value into callbacks (accepted args defaults to 1). This callback declares zero parameters, which can cause argument-count warnings/errors during the test run.
	public function filter_pre_get_col_charset_utf8mb4_for_encode_emoji_in_fields_test() {
		return 'utf8mb4';
	}

src/wp-includes/formatting.php:6125

  • The docblock says this encodes “emoji (and other 4-byte characters)”, but wp_encode_emoji() only converts emoji from the core emoji list; it does not generally encode arbitrary 4‑byte UTF‑8 characters. This wording should be narrowed to avoid overstating behavior.
 * Used by wp_insert_post(), wp_insert_comment(), and wp_update_comment() so
 * that emoji (and other 4-byte characters) are stored as HTML entities on
 * `utf8`/`utf8mb3` columns instead of causing truncated or failed writes.

bazza and others added 11 commits July 23, 2026 23:03
…-comments

# Conflicts:
#	tests/phpunit/tests/comment.php
Document that the helper encodes recognized emoji and add a TODO for all four-byte UTF-8 characters.
Use lowercase hexadecimal character references for all four-byte UTF-8 characters. This preserves supplementary characters outside the emoji list and leaves characters that fit in utf8mb3 intact. Match valid sequences even when the field also contains malformed bytes.
Check whether the encoded field fits before stripping the submitted text. This lets the retry clean an unrelated field while passing the original characters to moderation filters. Fields whose stored form needs cleanup still use the existing retry behavior.
…rences.

Encode literal emoji before checking for hexadecimal character references. In utf8mb3 posts, a family can contain four-byte characters stored as references and three-byte joiners stored literally. Render the family as one image. A separate reference in utf8mb4 content also permits literal emoji to render as images.

This change accompanies the four-byte storage encoder in ff855fd. If the PR uses wp_encode_emoji() for storage again, both changes can be reverted together.
@sirreal

sirreal commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

This work has been prioritized for 7.2. I've worked to get it into a state where I think it can be merged. I won't be available for the next week to continue working on this.

This PR now rewrites valid four-byte UTF-8 characters as HTML character references in comment authors and content when the database column uses utf8mb3. Posts and comments use the same encoder under the same conditions. 1, 2, and 3-byte UTF-8 is left alone.

Some considerations:

Previously, wp_staticize_emoji() skipped encoding literal emoji whenever the input contained &#x. On utf8mb3, 👨‍👩‍👧 is stored with character references for the three people and literal joiners between them. The old renderer produced three images. It now produces one family image. This also changes 😀 &#x41; on utf8mb4: the smile becomes an image and &#x41; is preserved.

If the UTF-8 4-byte HTML encoding is considered too invasive for this point, one option is to restore the emoji-only replacement. Revert 032e0e90d8 and ff855fdc6d and restore the is_string() guard introduced in these commits. This would replace emoji, including some 3-byte sequences, and leaves other 4-byte UTF-8 untouched.

One point to consider for review: this is not HTML aware. This type of encoding is not applicable everywhere in HTML. Notably tag and attribute names do not support this encoding, nor do some element contents like HTML script or style. That was already a limitation of the way the post content is handled. The content could be processed in an HTML-aware way, but that's a much more invasive change and it's difficult to implement correctly at this time (see the KSES Tag Processor implementation for some of the complexity involved). Given that the content cannot be stored or would be mangled with the 4-byte characters, the HTML encoding seems very reasonable to me.

For example:

<!-- these are equivalent -->
<div class="🙂">🙂</div>
<div class="&#x1f642;">&#x1f642;</div>

<!-- these are not equivalent -->
<custom-tag-🙂-smiley data-smiley-🙂-attr></custom-tag-🙂-smiley>
<custom-tag-&#x1f642;-smiley data-smiley-&#x1f642;-attr></custom-tag-&#x1f642;-smiley>

Keep the submitted field limits and check the encoded value against the column capacity. Reject a filtered value that exceeds the column before the insert retry can strip four-byte characters.
Keep the existing wp_insert_post() encoding for titles, content, and excerpts. Remove the tests added for the broader post encoding.
Use the actual column charsets so the test can only run where both comment fields can store four-byte characters.
Apply the storage length check in the insert retry only to author or content changed by four-byte character encoding. The retry still truncates overlong plain text and URLs.
@sirreal sirreal changed the title Comments: Encode emoji for utf8mb3 comment columns in wp_insert_comme… Comments: Encode emoji for utf8mb3 comment columns Nov 19, 2026
Prepare the author and content before the existing length checks. Use the same validator for encoded fields during the insert retry so the configured limits and field-specific errors apply in both places.
@sirreal

sirreal commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

I've narrowed this PR for 7.2. It now encodes four-byte characters as HTML character references in comment authors and content on utf8 and utf8mb3 columns. I've left broader changes to post encoding for a later release.

I reverted the change to wp_staticize_emoji(). In HTML email and RSS feeds, a family emoji such as 👨‍👩‍👧 can render as three separate images when the comment is stored on utf8mb3. I'd accept that limitation for 7.2 given how late we are in the release cycle.

I think this can ship in 7.2. I'd appreciate a closer review of the comment length checks, where I spent the least review time. The two cases I'd focus on are how the existing field limits apply after encoding expands the author or content, and how the insert retry handles a value that a comment filter has made too long.

I'll hand off the work now.

@dmsnell dmsnell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my note is for myself to try and come back later to today, but it’s free for anyone who wants to consider it in the meantime.

the one major difference using mb_encode_numericentity() is that WordPress has no polyfill of it, unlike how it does for mb_ord(). however, it would not be onerous to add one as it would build on top of _wp_scan_utf8()

| \xF4 [\x80-\x8F] [\x80-\xBF] [\x80-\xBF] # U+100000..U+10FFFF
/x',
static function ( $match ) {
return sprintf( '&#x%x;', mb_ord( $match[0], 'UTF-8' ) );

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a pre-emptive wp_scrub_utf8() will ensure we only have valid UTF-8, and then we can use mb_encode_numericentity() to perform the replacement within the C code. I haven’t measured this, but I am assuming it’s faster.

$data[ $field ] = mb_encode_numericentity(
	$data[ $field ],
	array( 0x10000, 0x10FFFF, 0, 0x1FFFF ),
	'UTF-8',
	true // use hexadecimal numeric character references
);

in fact, if we are careful to set and reset mb_substitude_character() properly then this call will also replace invalid byte sequences according to the maximal-subpart rule. so we could skip wp_scrub_utf8() if we set mb_substitute_character( 0xFFFD ) and then reset it when done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants