Skip to content

Fix IntCastingNaNError when repairing Volume with NaN in sudden-change ranges#2884

Open
8910work-cell wants to merge 1 commit into
ranaroussi:devfrom
8910work-cell:fix/nan-volume-sudden-change-repair
Open

Fix IntCastingNaNError when repairing Volume with NaN in sudden-change ranges#2884
8910work-cell wants to merge 1 commit into
ranaroussi:devfrom
8910work-cell:fix/nan-volume-sudden-change-repair

Conversation

@8910work-cell

Copy link
Copy Markdown

Fixes #2855.

Problem

_fix_prices_sudden_change() rounds and casts corrected Volume values to int in four places. Only the last of the four (at the end of the function) guards against NaN:

if correct_volume:
    f_na = df2['Volume'].isna()
    if f_na.any():
        df2.loc[~f_na,'Volume'] = df2['Volume'][~f_na].round(0).astype('int')
    else:
        df2['Volume'] = df2['Volume'].round(0).astype('int')

The other three call sites do .round().astype('int') directly with no such guard. If a Volume value inside a detected price-repair range (e.g. a bad stock split) is NaN, this raises pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer and aborts the whole repair — matching the traceback in #2855.

Fix

Add a small _round_volume_nansafe() helper that mirrors the NaN-tolerant pattern already used at the end of the function (round; cast to int only if no NaN is present, otherwise leave as rounded floats with NaN preserved), and use it at all four call sites instead of the bare .round().astype('int').

Verification

Reused the checked-in 4063.T bad-stock-split fixture (tests/data/4063-T-1d-bad-stock-split*.csv), which is already known to trigger a Volume-correcting repair range (see TestPriceRepairAssumptions.test_repair_bad_stock_splits). Injecting a NaN into a row inside that corrected range:

  • Before this fix: raises IntCastingNaNError (matches the reported bug, confirmed on current dev)
  • After this fix: no longer raises; the NaN is preserved rather than silently coerced into a wrong integer
  • Clean data (no NaN): repair output is unchanged and still matches the existing known-good fixture exactly — no regression

Adds two network-free tests to tests/test_price_repair.py (TestFixPricesSuddenChangeVolumeNaN) covering both cases.

…e ranges

_fix_prices_sudden_change() rounds and casts corrected Volume values to
int in four places. Only the last of the four (at the end of the
function) guards against NaN; the other three call `.round().astype('int')`
directly, so a NaN Volume falling inside a detected price-repair range
(e.g. a bad stock split) raises pandas.errors.IntCastingNaNError and
aborts the whole repair.

Add a small `_round_volume_nansafe()` helper - matching the NaN-tolerant
pattern already used at the end of the function - and use it at all four
call sites instead of the bare `.round().astype('int')`.

Fixes ranaroussi#2855

Verified against the existing '4063.T' bad-stock-split fixture: injecting
a NaN into a row inside a known-corrected range reproduces
IntCastingNaNError on current dev (red), and no longer raises after this
fix (green) while preserving the NaN. Clean data (no NaN) still matches
the pre-existing known-good fixture exactly - no regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ValueRaider

Copy link
Copy Markdown
Collaborator

Maybe switching type to nullable int is better https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html

@8910work-cell

Copy link
Copy Markdown
Author

Agreed that nullable Int64 is the "correct" dtype for a Volume that can be NA, and I looked into it. The blocker is that these are in-place partial assignments into an existing column, e.g.:

df2.loc[f_open_and_closed_fixed, "Volume"] = (df2.loc[f_open_and_closed_fixed, "Volume"] * m_rcp).round().astype('int')

On pandas 3.0, assigning an Int64 series that contains <NA> into a slice of a non-nullable int64/float64 column raises LossySetitemErrorTypeError: Invalid value '<NA>' for dtype .... The column has to already be nullable for the partial write to accept NA, so using Int64 here would mean migrating the whole Volume column to Int64 across the repair pipeline — which changes the dtype every downstream consumer of the repaired frame sees. That's a much wider blast radius than this crash fix.

I also kept it consistent with the NaN handling already in this same function a few lines below, which leaves NaN rows as floats rather than casting:

if correct_volume:
    f_na = df2['Volume'].isna()
    if f_na.any():
        df2.loc[~f_na, 'Volume'] = df2['Volume'][~f_na].round(0).astype('int')
    else:
        df2['Volume'] = df2['Volume'].round(0).astype('int')

_round_volume_nansafe does exactly the same thing (int when clean, rounded float when a NaN is present), so the fix stays minimal and local to the crash. Happy to do the full Int64 migration as a separate PR if you'd prefer Volume to become nullable-int project-wide — just let me know which direction you'd like.

@ValueRaider

Copy link
Copy Markdown
Collaborator

migrating the whole Volume column to Int64

Yes. It shouldn't cause any harm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants