Skip to content

BUG: assert_series_equal wrong with check_category_order=False and NAs (GH#62008) - #67723

Open
jbrockmendel wants to merge 1 commit into
pandas-dev:mainfrom
jbrockmendel:bug-62008
Open

BUG: assert_series_equal wrong with check_category_order=False and NAs (GH#62008)#67723
jbrockmendel wants to merge 1 commit into
pandas-dev:mainfrom
jbrockmendel:bug-62008

Conversation

@jbrockmendel

Copy link
Copy Markdown
Member

closes #62008

Missing values have code -1, and Index.take(-1) without allow_fill wraps around to the last category. assert_categorical_equal compared values via left.categories.take(left.codes), so two Categoricals holding equal values but storing their categories in a different order resolved the NA position to different labels and compared unequal:

values = ["B", np.nan, "D"]
left = pd.Series(pd.Categorical(values, categories=["B", "D"]))
right = pd.Series(pd.Categorical(values, categories=["D", "B"]))
pd.testing.assert_series_equal(left, right, check_category_order=False)
# AssertionError: Series category.values are different

This recodes both sides against the sorted categories and compares the codes, so -1 stays -1 and can only match -1.

Passing allow_fill=True, fill_value=np.nan to take — the approach in the closed GH-62011, GH-62017 and GH-66116 — fixes the reported case but casts integer categories to float64, so 2**53 and 2**53 + 1 collapse into one value and genuinely different Categoricals start comparing equal. There is a regression test pinning that.

Scope note: only assert_series_equal is affected on the public surface. assert_frame_equal and assert_index_equal do not accept or forward check_category_order, and the converse failure (unequal objects comparing equal) is not reachable through assert_series_equal, which compares values before it reaches assert_categorical_equal.

While here: check_category_order=False is still silently ignored when the categories cannot be sorted (mixed types), because the except TypeError fallback compares the categories in their original order. That is pre-existing and untouched here; the new tests pin the current behavior.

AI disclosure: drafted with Claude Code (claude opus 5 (high)), which wrote the fix and tests and ran the test suite; reviewed and revised by me.

…s (GH#62008)

Missing values have code -1, and Index.take(-1) without allow_fill wraps
around to the last category.  Two Categoricals holding equal values but
storing their categories in a different order therefore resolved the NA
position to different labels and compared unequal.

Compare the codes recoded against the sorted categories instead.  Filling
the taken values with NaN would work for the reported case but casts
integer categories to float64, so 2**53 and 2**53 + 1 would collapse into
one value and genuinely different Categoricals would compare equal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jbrockmendel jbrockmendel added Bug Testing pandas testing functions or related to the test suite Categorical Categorical Data Type labels Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Categorical Categorical Data Type Testing pandas testing functions or related to the test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: assert_series_equal broken with check_category_order=False for arrays with null values

1 participant