Repository navigation
Conversation
…nt() and wp_update_comment(). wp_insert_post() already encodes emoji as HTML entities before saving to a utf8/utf8mb3 column, via wp_encode_emoji(). wp_insert_comment() and wp_update_comment() had no equivalent handling. Extracts the shared logic into a new private helper, _wp_encode_emoji_in_fields( $data, $table, $fields ), in wp-includes/formatting.php, used by wp_insert_post() (refactor, no behavior change) and now also by wp_insert_comment() and wp_update_comment() for the comment_author and comment_content fields. Includes PHPUnit coverage for the new helper directly, plus deterministic (pre_get_col_charset-filter-forced) tests for wp_insert_post(), wp_insert_comment(), and wp_update_comment(), so the utf8mb3 code path is exercised in CI regardless of the test database's actual charset. Props barry. Fixes #65700.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @barry@a8c.com. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
|
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
There was a problem hiding this comment.
Pull request overview
This PR ensures emoji are safely stored when writing to utf8/utf8mb3 comment columns by introducing a shared helper to HTML-entity encode emoji for legacy 3-byte UTF-8 DB columns, aligning comment insertion/update behavior with the existing post insertion logic.
Changes:
- Extracts shared emoji-encoding logic into
_wp_encode_emoji_in_fields()(private helper) and refactorswp_insert_post()to use it (intended no behavior change). - Applies the helper to
wp_insert_comment()andwp_update_comment()forcomment_authorandcomment_content. - Adds PHPUnit coverage for the helper and deterministic tests that force the utf8mb3 code path via
pre_get_col_charset.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/phpunit/tests/post.php | Adds a deterministic utf8mb3-path test for wp_insert_post() via pre_get_col_charset. |
| tests/phpunit/tests/formatting/emoji.php | Adds direct unit tests for _wp_encode_emoji_in_fields() behavior and charset alias handling. |
| tests/phpunit/tests/comment.php | Adds deterministic tests for utf8mb3-path encoding in wp_insert_comment() / wp_update_comment(), plus a utf8mb4 control. |
| src/wp-includes/post.php | Refactors emoji encoding in wp_insert_post() to use the new shared helper. |
| src/wp-includes/formatting.php | Introduces _wp_encode_emoji_in_fields() helper used across post/comment insert/update flows. |
| src/wp-includes/comment.php | Applies emoji encoding helper to comment insert/update for legacy utf8/utf8mb3 columns. |
Comments suppressed due to low confidence (1)
tests/phpunit/tests/formatting/emoji.php:458
- This filter callback is invoked via
pre_get_col_charsetand will be called with arguments; keeping a 0-arg signature can emit warnings. Update the signature to accept the standard$charset, $table, $columnparameters (even if unused) so it’s compatible with the hook invocation.
public function filter_pre_get_col_charset_utf8_alias_for_encode_emoji_in_fields_test() {
return 'utf8';
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… hitting a real DB query. The test passed a fake 'some_table' table name to _wp_encode_emoji_in_fields() without short-circuiting wpdb::get_col_charset() via the pre_get_col_charset filter, unlike the other tests in this file. This caused a real 'SHOW FULL COLUMNS FROM some_table' query against a table that doesn't exist, failing with a WordPress database error instead of testing the intended behavior (that a field missing from $data is left alone). See #65700.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
tests/phpunit/tests/formatting/emoji.php:465
- The
pre_get_col_charsetfilter always passes at least the$charsetargument (accepted args defaults to 1 when omitted). These callbacks declare zero parameters, which can trigger “too many arguments” warnings/errors under PHPUnit/PHP and make the test suite fail.
public function filter_pre_get_col_charset_utf8_alias_for_encode_emoji_in_fields_test() {
return 'utf8';
}
tests/phpunit/tests/formatting/emoji.php:469
- The
pre_get_col_charsetfilter passes the$charsetvalue into callbacks (accepted args defaults to 1). This callback declares zero parameters, which can cause argument-count warnings/errors during the test run.
public function filter_pre_get_col_charset_utf8mb4_for_encode_emoji_in_fields_test() {
return 'utf8mb4';
}
src/wp-includes/formatting.php:6125
- The docblock says this encodes “emoji (and other 4-byte characters)”, but
wp_encode_emoji()only converts emoji from the core emoji list; it does not generally encode arbitrary 4‑byte UTF‑8 characters. This wording should be narrowed to avoid overstating behavior.
* Used by wp_insert_post(), wp_insert_comment(), and wp_update_comment() so
* that emoji (and other 4-byte characters) are stored as HTML entities on
* `utf8`/`utf8mb3` columns instead of causing truncated or failed writes.
…-comments # Conflicts: # tests/phpunit/tests/comment.php
Document that the helper encodes recognized emoji and add a TODO for all four-byte UTF-8 characters.
Use lowercase hexadecimal character references for all four-byte UTF-8 characters. This preserves supplementary characters outside the emoji list and leaves characters that fit in utf8mb3 intact. Match valid sequences even when the field also contains malformed bytes.
Check whether the encoded field fits before stripping the submitted text. This lets the retry clean an unrelated field while passing the original characters to moderation filters. Fields whose stored form needs cleanup still use the existing retry behavior.
…rences. Encode literal emoji before checking for hexadecimal character references. In utf8mb3 posts, a family can contain four-byte characters stored as references and three-byte joiners stored literally. Render the family as one image. A separate reference in utf8mb4 content also permits literal emoji to render as images. This change accompanies the four-byte storage encoder in ff855fd. If the PR uses wp_encode_emoji() for storage again, both changes can be reverted together.
|
This work has been prioritized for 7.2. I've worked to get it into a state where I think it can be merged. I won't be available for the next week to continue working on this. This PR now rewrites valid four-byte UTF-8 characters as HTML character references in comment authors and content when the database column uses Some considerations: Previously, If the UTF-8 4-byte HTML encoding is considered too invasive for this point, one option is to restore the emoji-only replacement. Revert One point to consider for review: this is not HTML aware. This type of encoding is not applicable everywhere in HTML. Notably tag and attribute names do not support this encoding, nor do some element contents like HTML script or style. That was already a limitation of the way the post content is handled. The content could be processed in an HTML-aware way, but that's a much more invasive change and it's difficult to implement correctly at this time (see the KSES Tag Processor implementation for some of the complexity involved). Given that the content cannot be stored or would be mangled with the 4-byte characters, the HTML encoding seems very reasonable to me. For example: <!-- these are equivalent -->
<div class="🙂">🙂</div>
<div class="🙂">🙂</div>
<!-- these are not equivalent -->
<custom-tag-🙂-smiley data-smiley-🙂-attr></custom-tag-🙂-smiley>
<custom-tag-🙂-smiley data-smiley-🙂-attr></custom-tag-🙂-smiley> |
Keep the submitted field limits and check the encoded value against the column capacity. Reject a filtered value that exceeds the column before the insert retry can strip four-byte characters.
…ter references." This reverts commit 032e0e9.
Keep the existing wp_insert_post() encoding for titles, content, and excerpts. Remove the tests added for the broader post encoding.
Use the actual column charsets so the test can only run where both comment fields can store four-byte characters.
Apply the storage length check in the insert retry only to author or content changed by four-byte character encoding. The retry still truncates overlong plain text and URLs.
Prepare the author and content before the existing length checks. Use the same validator for encoded fields during the insert retry so the configured limits and field-specific errors apply in both places.
|
I've narrowed this PR for 7.2. It now encodes four-byte characters as HTML character references in comment authors and content on I reverted the change to I think this can ship in 7.2. I'd appreciate a closer review of the comment length checks, where I spent the least review time. The two cases I'd focus on are how the existing field limits apply after encoding expands the author or content, and how the insert retry handles a value that a comment filter has made too long. I'll hand off the work now. |
dmsnell
left a comment
There was a problem hiding this comment.
my note is for myself to try and come back later to today, but it’s free for anyone who wants to consider it in the meantime.
the one major difference using mb_encode_numericentity() is that WordPress has no polyfill of it, unlike how it does for mb_ord(). however, it would not be onerous to add one as it would build on top of _wp_scan_utf8()
| | \xF4 [\x80-\x8F] [\x80-\xBF] [\x80-\xBF] # U+100000..U+10FFFF | ||
| /x', | ||
| static function ( $match ) { | ||
| return sprintf( '&#x%x;', mb_ord( $match[0], 'UTF-8' ) ); |
There was a problem hiding this comment.
a pre-emptive wp_scrub_utf8() will ensure we only have valid UTF-8, and then we can use mb_encode_numericentity() to perform the replacement within the C code. I haven’t measured this, but I am assuming it’s faster.
$data[ $field ] = mb_encode_numericentity(
$data[ $field ],
array( 0x10000, 0x10FFFF, 0, 0x1FFFF ),
'UTF-8',
true // use hexadecimal numeric character references
);in fact, if we are careful to set and reset mb_substitude_character() properly then this call will also replace invalid byte sequences according to the maximal-subpart rule. so we could skip wp_scrub_utf8() if we set mb_substitute_character( 0xFFFD ) and then reset it when done.
On
utf8andutf8mb3columns, comment writes can fail or lose four-byte UTF-8 characters. Preserve those characters in comment authors and content by storing them as HTML character references. Duplicate detection, previously approved author checks, andcomment_exists(), and length limit checks use the same encoding as writes.Trac ticket: https://core.trac.wordpress.org/ticket/65700
See also: https://core.trac.wordpress.org/ticket/46947
Use of AI Tools
AI assistance: Yes
Tool(s): Claude, Codex
Model(s): Sonnet, Fable, GPT…
Used for: Code abstraction and tests. Human reviewed and tested.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.