Skip to content

wr.postgresql.read_sql_query() - KeyError when reading UUID column values from Postgres #3338

Description

@rdwebster

Describe the bug

After upgrading awswrangler, I started getting errors when reading values of UUID columns (datatype 1111) in Postgres.
The error happens with awswrangler-3.16.1 (and pyarrow-24.0.0), but does not happen with awswrangler-3.16.0 (and pyarrow-23.0.1).

How to Reproduce

Example code:

import awswrangler as wr
import pg8000

con = pg8000.connect(
    user="test_user",
    password="???",
    host="localhost",
    port=5432,
    database="test_database"
)

sql = "SELECT uuid_column FROM test.uuid_test_table LIMIT 10"

try:
    df = wr.postgresql.read_sql_query(sql=sql, con=con)
    print("Query succeeded:")
    print(df)
except Exception as e:
    print(f"Error occurred: {type(e).__name__}: {e}")
finally:
    con.close()

ERROR:
Error occurred: KeyError: 'uuid_column'

Full traceback:

Traceback (most recent call last):
  File "/.../reproduce_uuid_issue.py", line 24, in <module>
    df = wr.postgresql.read_sql_query(sql=sql, con=con)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/awswrangler/_utils.py", line 126, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/awswrangler/postgresql.py", line 360, in read_sql_query
    return _db_utils.read_sql_query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/awswrangler/_databases.py", line 330, in read_sql_query
    return _fetch_all_results(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/awswrangler/_databases.py", line 260, in _fetch_all_results
    return _records2df(
           ^^^^^^^^^^^^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/awswrangler/_databases.py", line 182, in _records2df
    df = table.to_pandas(  # Creating Pandas DataFrame
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/array.pxi", line 1071, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 5146, in pyarrow.lib.Table._to_pandas
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 827, in table_to_dataframe
    blocks = [
             ^
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 828, in <listcomp>
    _reconstruct_block(
  File "/opt/miniconda3/envs/test_proj/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 773, in _reconstruct_block
    pandas_dtype = extension_columns[name]
                   ~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'uuid_column'

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

Mac, Linux

Python version

3.11

AWS SDK for pandas version

3.16.1

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions