BUG: read_csv low_memory mixed-dtype warning named the wrong columns (GH#67375) - #67725
Open
Parth1353 wants to merge 1 commit into
Open
BUG: read_csv low_memory mixed-dtype warning named the wrong columns (GH#67375)#67725Parth1353 wants to merge 1 commit into
Parth1353 wants to merge 1 commit into
Conversation
…(GH#67375) read_low_memory keys its chunks by field position in the source row, but _concatenate_chunks used those positions to index the column names. The two only line up when nothing was dropped, so: - usecols raised IndexError for any selected column that had moved, which also reached the chunksize and iterator readers; - an index column shifted the positions past the data columns, silently naming the column after the mixed one. Map the positions to labels the way read() renames these same keys: skip the leading implicit index columns, then zip against orig_names, which usecols has already been applied to and which the reader has already deduplicated. The number beside each name was wrong too. GH#58174 asked for the existing "Columns (15,19)" positions to be kept and annotated with names; GH#58250 added the names but numbered them with enumerate over the warned columns, so the numbers became 0, 1, 2 ... while still looking like the positions they had replaced. Carry each column's real position through instead. A leading implicit index column has no name to report, so it stays a bare position rather than borrowing the name of the first data column.
Parth1353
force-pushed
the
fix-67375-low-memory-warning-columns
branch
from
September 1, 2026 17:24
ef04e0e to
fe4e34c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/v3.1.0.rstfileread_low_memorykeys its chunks by field position in the source row, but_concatenate_chunksused those positions to index the column-name list. The two onlyline up when nothing was dropped, so the mixed-dtype
DtypeWarningwas wrong in threeways — all from #58250, all shipped in 3.0.0, none present in 2.x:
usecolsraisedIndexErrorfor any selected column not still at its originalposition (the reported bug). It reaches
chunksize=anditerator=too, not justplain
read_csv.index_col=0a file whosemixed columns are
c3, c50, c80reportedc4, c51, c81. No crash, no clue.existing
Columns (15,19)positions and annotate them with names; ENH: DtypeWarning message enhancement #58250 numbered themwith
enumerateover the warned columns instead, so they became 0, 1, 2 … while stilllooking like the positions they replaced.
Fixed by mapping positions to labels the way
read()renames these same keys: skipleading_cols, then zip againstorig_names— whichusecolshas already been applied toand which the reader has already deduplicated.
namesis the wrong list; it drops indexcolumns.
A leading implicit index column has no name, so it stays a bare position rather than
borrowing the first data column's name. The mapping deliberately covers every parsed
position: a partial one turns a wrong label into a
KeyErrorwhen that unnamed indexcolumn is itself the mixed one.
Data was never affected — only the warning. Verified by cross-checking 28 option
combinations against
engine="python"at bothlow_memorysettings.If the message change in (3) is unwanted here, it is easy to split out; (1) and (2) stand
on their own.
AI disclosure: written with Claude Code (claude opus 5).