Skip to content

BUG: fix IndexError when usecols selects a column not at position 0 - #67626

Open
Himesh-rupchandani wants to merge 2 commits into
pandas-dev:mainfrom
Himesh-rupchandani:fix/67375-usecols-concatenate-indexerror
Open

BUG: fix IndexError when usecols selects a column not at position 0#67626
Himesh-rupchandani wants to merge 2 commits into
pandas-dev:mainfrom
Himesh-rupchandani:fix/67375-usecols-concatenate-indexerror

Conversation

@Himesh-rupchandani

Copy link
Copy Markdown

Closes GH#67375.

What

read_csv(usecols=...) with low_memory=True raised IndexError when a
usecols-selected column was at a position other than 0 and different chunks
inferred mixed dtypes for that column.

Root cause

In _concatenate_chunks, chunk keys are raw column positions (e.g. 50), but
column_names is already filtered by usecols, so column_names[name]
indexed by the raw position and went out of range. It only worked when the
column happened to be at position 0.

Fix

Build a position->name mapping from the sorted chunk keys (which align with
the ascending-position order of column_names) and look up by position
instead of positional index.

Test

Adds test_concatenate_chunks_usecols_mixed_dtype_nonzero_pos in
test_concatenate_chunks.py. It fails before the fix (IndexError) and passes
after.

AI assistance

I used an AI assistant to reproduce the bug, locate the cause, and draft the
fix and test; I reviewed all changed lines and verified the behaviour locally.

@Himesh-rupchandani

Copy link
Copy Markdown
Author

Hi, I've reproduced this and prepared a fix + regression test. PR is up.

Summary: in _concatenate_chunks the chunk keys are raw column positions but
column_names is already filtered by usecols, so column_names[name] indexes
out of range for a selected column at position > 0. Fix: map position->name
from the sorted chunk keys.

@github-actions github-actions Bot added the Awaiting Review PR in the maintainers' court; exempt from staleness label Sep 1, 2026
read_csv with low_memory=True and usecols selecting a column whose position
is not 0 raised IndexError when different chunks inferred mixed dtypes for
that column.

In _concatenate_chunks the chunk keys are raw column positions, but
column_names is already filtered by usecols, so it cannot be indexed by the
raw position directly. Map position -> name from the sorted chunk keys
(aligned with the ascending-position order of column_names) and look up by
position instead of by positional index.

Adds a regression test that reproes the failure at the _concatenate_chunks
level (mixed-dtype warning for a column at position 50).

Closes GH#67375

Co-authored-by: Himesh Rupchandani <rupchandanihimesh69@gmail.com>
@Himesh-rupchandani
Himesh-rupchandani force-pushed the fix/67375-usecols-concatenate-indexerror branch from fd95694 to 7d1971b Compare September 1, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Awaiting Review PR in the maintainers' court; exempt from staleness

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant