Skip to content

SNOW-3243341: python wrapper - support pandas and pyarrow fetching#653

Merged
sfc-gh-asolarski merged 1 commit intomainfrom
03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching
Mar 26, 2026
Merged

SNOW-3243341: python wrapper - support pandas and pyarrow fetching#653
sfc-gh-asolarski merged 1 commit intomainfrom
03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching

Conversation

@sfc-gh-asolarski
Copy link
Copy Markdown
Contributor

@sfc-gh-asolarski sfc-gh-asolarski commented Mar 20, 2026 •
edited
Loading

TL;DR

Added support for Arrow/Pandas fetch methods (fetch_arrow_all, fetch_arrow_batches, fetch_pandas_all, fetch_pandas_batches) to the Snowflake Python connector by implementing a new stream-based Arrow table iterator with Snowflake-specific type conversions.

What changed?

  • New C++ Components: Added ArrowTableConverter class to handle Snowflake-specific type conversions (scaled numbers, timestamps, etc.) and CArrowStreamTableIterator for stream-based Arrow batch processing
  • Python Interface: Implemented fetch_arrow_all, fetch_arrow_batches, fetch_pandas_all, and fetch_pandas_batches methods in the cursor class
  • Type Safety: Added fetch mode validation to prevent mixing row-by-row and Arrow/Pandas fetching within the same result set
  • Connection Properties: Added arrow_number_to_decimal property to control numeric type conversion behavior
  • Build System: Updated build configuration to include new C++ source files

This was referenced Mar 20, 2026
Merged
Merged
@sfc-gh-asolarski sfc-gh-asolarski marked this pull request as ready for review March 20, 2026 11:12
@sfc-gh-asolarski sfc-gh-asolarski mentioned this pull request Mar 20, 2026
Merged
@sfc-gh-asolarski sfc-gh-asolarski requested a review from a team as a code owner March 20, 2026 11:12
This was referenced Mar 20, 2026
Merged
Merged
@sfc-gh-asolarski Graphite App
Copy link
Copy Markdown
Contributor Author

sfc-gh-asolarski commented Mar 20, 2026 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

@sfc-gh-asolarski sfc-gh-asolarski mentioned this pull request Mar 20, 2026
Merged
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch 2 times, most recently from bf54370 to ba1f2bc Compare March 20, 2026 13:10
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ branch from 4d0bc36 to b4692bf Compare March 20, 2026 13:10
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from ba1f2bc to b593112 Compare March 20, 2026 13:21
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ branch 2 times, most recently from ce4fb08 to 1e58685 Compare March 20, 2026 14:34
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from b593112 to 2ff7ed5 Compare March 20, 2026 14:34
sfc-gh-turbaszek
sfc-gh-turbaszek reviewed Mar 23, 2026
View reviewed changes
python/src/snowflake/connector/_internal/nanoarrow_cpp/ArrowIterator/ArrowTableConverter.cpp
@@ -0,0 +1,870 @@
#include "ArrowTableConverter.hpp"
Copy link
Copy Markdown
Contributor

@sfc-gh-turbaszek sfc-gh-turbaszek Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this copy past or new implementation?

Copy link
Copy Markdown
Contributor Author

@sfc-gh-asolarski sfc-gh-asolarski Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract from old nanoarrow CArrowStreamTableIterator

Copy link
Copy Markdown
Contributor Author

@sfc-gh-asolarski sfc-gh-asolarski Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CArrowStreamTableIterator is new

sfc-gh-turbaszek
sfc-gh-turbaszek requested changes Mar 23, 2026
View reviewed changes
python/tests/e2e/pandas/test_fetch_arrow.py Outdated
Comment on lines +169 to +172
if is_old_driver():
cursor.connection.arrow_number_to_decimal_setter = True
else:
cursor.connection.arrow_number_to_decimal = True
Copy link
Copy Markdown
Contributor

@sfc-gh-turbaszek sfc-gh-turbaszek Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not necessary, old driver supports arrow_number_to_decimal

Copy link
Copy Markdown
Contributor Author

@sfc-gh-asolarski sfc-gh-asolarski Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing is that there is a bug in the old driver (the method name of the setter is wrong), hence it must be used like this.

@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 2ff7ed5 to c014065 Compare March 23, 2026 11:53
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ branch from 1e58685 to c91a26d Compare March 23, 2026 11:53
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from c014065 to 7d64026 Compare March 23, 2026 14:03
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ branch from c91a26d to 7963d70 Compare March 23, 2026 14:03
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ to graphite-base/653 March 23, 2026 15:30
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 7d64026 to 7dc9897 Compare March 23, 2026 19:52
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from graphite-base/653 to 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ March 23, 2026 19:53
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 7dc9897 to 7d8a3ed Compare March 24, 2026 08:20
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ branch from 7ddb233 to e0b4b64 Compare March 24, 2026 08:20
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from 03-19-snow-3243341_optional_dependencies_pyarrow_pandas_ to graphite-base/653 March 24, 2026 09:34
@sfc-gh-asolarski sfc-gh-asolarski mentioned this pull request Mar 24, 2026
Merged
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 7d8a3ed to 7817b1c Compare March 24, 2026 13:48
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from graphite-base/653 to 03-24-snow-3243341_python_decorators_for_required_conditions_in_cursor March 24, 2026 13:48
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 7817b1c to acf2477 Compare March 24, 2026 13:50
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from 03-24-snow-3243341_python_decorators_for_required_conditions_in_cursor to graphite-base/653 March 24, 2026 14:25
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from acf2477 to dcdd93e Compare March 24, 2026 14:46
@sfc-gh-asolarski sfc-gh-asolarski changed the base branch from graphite-base/653 to 03-24-snow-3243341_python_decorators_for_required_conditions_in_cursor March 24, 2026 14:46
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from dcdd93e to 6af3814 Compare March 24, 2026 15:13
Base automatically changed from 03-24-snow-3243341_python_decorators_for_required_conditions_in_cursor to main March 24, 2026 15:42
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch 3 times, most recently from 9bf8999 to b2a1f1b Compare March 25, 2026 13:58
@sfc-gh-asolarski sfc-gh-asolarski mentioned this pull request Mar 25, 2026
Merged
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from b2a1f1b to 3fc02a0 Compare March 25, 2026 14:35
@sfc-gh-asolarski sfc-gh-asolarski mentioned this pull request Mar 25, 2026
Closed
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from 3fc02a0 to d1c0313 Compare March 26, 2026 11:33
@sfc-gh-asolarski
SNOW-3243341: python wrapper - support pandas and pyarrow fetching
ce0a78b 
# Conflicts:
#	python/src/snowflake/connector/cursor.py

# Conflicts:
#	python/tests/unit/test_connection.py
@sfc-gh-asolarski sfc-gh-asolarski force-pushed the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch from d1c0313 to ce0a78b Compare March 26, 2026 13:46
@sfc-gh-asolarski sfc-gh-asolarski added this pull request to the merge queue Mar 26, 2026
Merged via the queue into main with commit 449cc35 Mar 26, 2026
75 of 78 checks passed
@sfc-gh-asolarski sfc-gh-asolarski deleted the 03-20-snow-3243341_python_wrapper_-_support_pandas_and_pyarrow_fetching branch March 26, 2026 15:11
This was referenced Mar 26, 2026
Merged
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-turbaszek sfc-gh-turbaszek sfc-gh-turbaszek approved these changes

Assignees

No one assigned

Labels

python tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sfc-gh-asolarski @sfc-gh-turbaszek