Fix IntCastingNaNError when repairing Volume with NaN in sudden-change ranges#2884
Fix IntCastingNaNError when repairing Volume with NaN in sudden-change ranges#28848910work-cell wants to merge 1 commit into
Conversation
…e ranges
_fix_prices_sudden_change() rounds and casts corrected Volume values to
int in four places. Only the last of the four (at the end of the
function) guards against NaN; the other three call `.round().astype('int')`
directly, so a NaN Volume falling inside a detected price-repair range
(e.g. a bad stock split) raises pandas.errors.IntCastingNaNError and
aborts the whole repair.
Add a small `_round_volume_nansafe()` helper - matching the NaN-tolerant
pattern already used at the end of the function - and use it at all four
call sites instead of the bare `.round().astype('int')`.
Fixes ranaroussi#2855
Verified against the existing '4063.T' bad-stock-split fixture: injecting
a NaN into a row inside a known-corrected range reproduces
IntCastingNaNError on current dev (red), and no longer raises after this
fix (green) while preserving the NaN. Clean data (no NaN) still matches
the pre-existing known-good fixture exactly - no regression.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Maybe switching type to nullable int is better https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html |
|
Agreed that nullable df2.loc[f_open_and_closed_fixed, "Volume"] = (df2.loc[f_open_and_closed_fixed, "Volume"] * m_rcp).round().astype('int')On pandas 3.0, assigning an I also kept it consistent with the NaN handling already in this same function a few lines below, which leaves NaN rows as floats rather than casting: if correct_volume:
f_na = df2['Volume'].isna()
if f_na.any():
df2.loc[~f_na, 'Volume'] = df2['Volume'][~f_na].round(0).astype('int')
else:
df2['Volume'] = df2['Volume'].round(0).astype('int')
|
Yes. It shouldn't cause any harm. |
Fixes #2855.
Problem
_fix_prices_sudden_change()rounds and casts correctedVolumevalues tointin four places. Only the last of the four (at the end of the function) guards against NaN:The other three call sites do
.round().astype('int')directly with no such guard. If aVolumevalue inside a detected price-repair range (e.g. a bad stock split) isNaN, this raisespandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integerand aborts the whole repair — matching the traceback in #2855.Fix
Add a small
_round_volume_nansafe()helper that mirrors the NaN-tolerant pattern already used at the end of the function (round; cast tointonly if no NaN is present, otherwise leave as rounded floats with NaN preserved), and use it at all four call sites instead of the bare.round().astype('int').Verification
Reused the checked-in
4063.Tbad-stock-split fixture (tests/data/4063-T-1d-bad-stock-split*.csv), which is already known to trigger a Volume-correcting repair range (seeTestPriceRepairAssumptions.test_repair_bad_stock_splits). Injecting aNaNinto a row inside that corrected range:IntCastingNaNError(matches the reported bug, confirmed on currentdev)NaNis preserved rather than silently coerced into a wrong integerAdds two network-free tests to
tests/test_price_repair.py(TestFixPricesSuddenChangeVolumeNaN) covering both cases.