Fix: roundtrip inconsistency for unqualified emoji (adds VS16)

[gaoflow]

Summary

Unqualified emoji entries (status=4) — e.g. U+2764 (❤) without U+FE0F — duplicate their fully-qualified counterpart under the same shortcode name. When demojize() processes such a form and the result is passed back to emojize(), the library silently adds a variation selector-16 that was not present in the original input.

Before (broken roundtrip):

>>> emoji.emojize(emoji.demojize("❤"))  # input: U+2764
"❤️"  # output: U+2764 U+FE0F — VS16 was not in the input

After (consistent):

>>> emoji.emojize(emoji.demojize("❤"))
"❤"   # unchanged — no spurious VS16

Root cause

EMOJI_DATA contains both the canonical fully-qualified (status=2) and the unqualified (status=4) form for 243 emoji, keyed by different Unicode strings but sharing the same shortcode name:

• EMOJI_DATA["\\U0001F170"] → :A_button_(blood_type): (status=4, no VS16)
• EMOJI_DATA["\\U0001F170\\U0000FE0F"] → :A_button_(blood_type): (status=2, with VS16)

demojize() happily converts either form to the same shortcode. emojize() always returns the fully-qualified version, so the roundtrip adds VS16 when none was there.

Fix

Filter out unqualified (status=4) entries from EMOJI_DATA at load time (in _load_default_from_json). These entries are non-canonical duplicates that are explicitly labelled as not recommended for general interchange by Unicode.org. Language JSON files are handled by skipping keys that are no longer in EMOJI_DATA.

• 12 insertions, 1 deletion — minimal, targeted change
• All 102 existing tests pass unchanged
• 243 previously non-roundtripping emoji now roundtrip correctly

Affected emoji: © ® ™ ♥ ♣ ♦ ♠ ♻ ☯ ✈ ❤ … (243 total)

AI disclosure: This PR was prepared under my direction with assistance from AI.

[gemini-code-assist]

Code Review

This pull request filters out unqualified emoji entries (status < 4) from the default emoji dataset to achieve roundtrip consistency. However, this change introduces a critical regression by breaking demojize() and replace_emoji() for many common unqualified emojis (such as the plain red heart). The feedback recommends reverting this filtering, as well as the corresponding check in the language-loading logic, to preserve core library functionality.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

☹ Critical Regression: Breaking demojize() and replace_emoji() for Unqualified Emojis

Filtering out unqualified (status=4) entries from EMOJI_DATA at load time introduces a major regression in the library's core functionality.

1. The Issue

demojize() and replace_emoji() rely on EMOJI_DATA to detect emojis in input text. By completely removing unqualified emojis from EMOJI_DATA, the library will no longer recognize them.

For example:

• Before this PR: emoji.demojize("I ❤ you") -> "I :red_heart: you" (U+2764 is successfully recognized and converted).
• After this PR: emoji.demojize("I ❤ you") -> "I ❤ you" (U+2764 is ignored and remains unconverted).

This affects 243 common emojis, including extremely prevalent characters like:

• ❤ (red heart, U+2764)
• © (copyright, U+00A9)
• ® (registered, U+00AE)
• ™ (trademark, U+2122)
• ⚠ (warning, U+26A0)

Since most users type or copy-paste these plain/unqualified versions without the variation selector-16 (\ufe0f), demojize() will fail to work for the vast majority of real-world inputs containing these characters.

2. Why the Roundtrip "Inconsistency" is Expected Behavior

The roundtrip "inconsistency" (emojize(demojize("❤")) returning "❤️") is actually the mathematically correct and expected behavior of a lossy shortcode representation.

Because both the unqualified form ("❤") and the fully-qualified form ("❤️") map to the exact same shortcode (:red_heart:), the qualification status is lost during demojize(). When converting back via emojize(), the library must choose a single default representation, which should always be the fully-qualified form as recommended by the Unicode Consortium for general interchange.

3. Recommendation

Revert this filtering entirely to preserve the ability to demojize and replace unqualified emojis.

Suggested change

	data = dict(json.load(f, object_pairs_hook=EmojiDataDict)) # type: ignore
	# Exclude unqualified (status=4) entries to ensure
	# emojize(demojize(s)) == s for all fully-qualified emoji.
	# These are non-canonical forms (e.g., bare U+2764 without U+FE0F)
	# that duplicate a fully-qualified entry with the same name.
	EMOJI_DATA = {
	k: v for k, v in data.items() if v.get('status', 0) < STATUS['unqualified']
	}
	EMOJI_DATA = dict(json.load(f, object_pairs_hook=EmojiDataDict)) # type: ignore

[gemini-code-assist]

This check is only necessary because of the unqualified emoji filtering in _load_default_from_json(). Once that filtering is reverted, this check should also be removed to ensure all language translations are correctly loaded for all emojis.

Suggested change

	# After filtering unqualified entries from _load_default_from_json(),
	# some keys from language JSON files no longer exist in EMOJI_DATA.
	if emj not in EMOJI_DATA:
	continue
	EMOJI_DATA[emj][key] = value # type: ignore
	EMOJI_DATA[emj][key] = value # type: ignore

[cvzi]

As mentioned in #328 I would prefer a simple change in the docstring to explain the current behaviour. If you need the roundtrip function, then I would suggest to create a new function specifically for this behaviour that doesn't modify the current behaviour and also doesn't modify the global EMOJI_DATA

[gaoflow]

Understood — mutating the global EMOJI_DATA is too invasive for a default-path change, that's fair. I'd rather not leave a rejected approach sitting open, so feel free to close this. If a separate opt-in roundtrip helper is something you'd take, I can open that on its own and leave demojize/emojize untouched.