Skip to content

BUG: read_csv low_memory mixed-dtype warning named the wrong columns (GH#67375) - #67725

Open
Parth1353 wants to merge 1 commit into
pandas-dev:mainfrom
Parth1353:fix-67375-low-memory-warning-columns
Open

BUG: read_csv low_memory mixed-dtype warning named the wrong columns (GH#67375)#67725
Parth1353 wants to merge 1 commit into
pandas-dev:mainfrom
Parth1353:fix-67375-low-memory-warning-columns

Conversation

@Parth1353

Copy link
Copy Markdown
Contributor

read_low_memory keys its chunks by field position in the source row, but
_concatenate_chunks used those positions to index the column-name list. The two only
line up when nothing was dropped, so the mixed-dtype DtypeWarning was wrong in three
ways — all from #58250, all shipped in 3.0.0, none present in 2.x:

  1. usecols raised IndexError for any selected column not still at its original
    position (the reported bug). It reaches chunksize= and iterator= too, not just
    plain read_csv.
  2. An index column silently named the wrong column. With index_col=0 a file whose
    mixed columns are c3, c50, c80 reported c4, c51, c81. No crash, no clue.
  3. The number beside each name was a counter, not a position. ENH: DtypeWarning message enhancement #58174 asked to keep the
    existing Columns (15,19) positions and annotate them with names; ENH: DtypeWarning message enhancement #58250 numbered them
    with enumerate over the warned columns instead, so they became 0, 1, 2 … while still
    looking like the positions they replaced.

Fixed by mapping positions to labels the way read() renames these same keys: skip
leading_cols, then zip against orig_names — which usecols has already been applied to
and which the reader has already deduplicated. names is the wrong list; it drops index
columns.

A leading implicit index column has no name, so it stays a bare position rather than
borrowing the first data column's name. The mapping deliberately covers every parsed
position: a partial one turns a wrong label into a KeyError when that unnamed index
column is itself the mixed one.

Data was never affected — only the warning. Verified by cross-checking 28 option
combinations against engine="python" at both low_memory settings.

If the message change in (3) is unwanted here, it is easy to split out; (1) and (2) stand
on their own.


AI disclosure: written with Claude Code (claude opus 5).

…(GH#67375)

read_low_memory keys its chunks by field position in the source row, but
_concatenate_chunks used those positions to index the column names. The two
only line up when nothing was dropped, so:

- usecols raised IndexError for any selected column that had moved, which also
  reached the chunksize and iterator readers;
- an index column shifted the positions past the data columns, silently naming
  the column after the mixed one.

Map the positions to labels the way read() renames these same keys: skip the
leading implicit index columns, then zip against orig_names, which usecols has
already been applied to and which the reader has already deduplicated.

The number beside each name was wrong too. GH#58174 asked for the existing
"Columns (15,19)" positions to be kept and annotated with names; GH#58250 added
the names but numbered them with enumerate over the warned columns, so the
numbers became 0, 1, 2 ... while still looking like the positions they had
replaced. Carry each column's real position through instead. A leading implicit
index column has no name to report, so it stays a bare position rather than
borrowing the name of the first data column.
@Parth1353
Parth1353 force-pushed the fix-67375-low-memory-warning-columns branch from ef04e0e to fe4e34c Compare September 1, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: read_csv with usecols raises IndexError in low-memory mixed-dtype warning path

1 participant