Skip to content

Timestamp resolution overflow with pandas>=3.0 #3357

Description

@robert-schmidtke

Describe the bug

Hi,

pandas 3.0 is defaulting timestamps to microsecond resolution where possible by default:
https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#datetime-timedelta-resolution-inference

While it was possible in pandas 2.0 already to use timestamps that have resolutions other than nanoseconds, the default change now makes it a more prominent change.

When writing and reading parquet files with small and large timestamps, only supported by using microsecond resolution, the results come back and look like some overflow happened in-between.

Note that the pd.Timestamp.min (1677-09-21 00:12:43.145224193) and pd.Timestamp.max (2262-04-11 23:47:16.854775807) "boundaries" in pandas still exist and they have nanosecond resolution.
It seems that dates in that range are working fine, but dates outside are not.

Somewhere along the way, nanosecond resolution must be enforced.

How to Reproduce

# Python 3.14.5 (main, May 10 2026, 19:28:16) [Clang 22.1.3 ] on linux
# Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import awswrangler as wr
>>> df = pd.DataFrame({"ts": pd.to_datetime(["1000-01-01 12:00:00", "1677-09-01 12:00:00", "1677-10-01 12:00:00", "2262-04-01 12:00:00", "2262-05-01 12:00:00", "3000-01-01 12:00:00"]), "value": [1000, 1677.1, 1677.2, 2262.1, 2262.2, 3000]})
>>>
>>> df
                   ts   value
0 1000-01-01 12:00:00  1000.0
1 1677-09-01 12:00:00  1677.1
2 1677-10-01 12:00:00  1677.2
3 2262-04-01 12:00:00  2262.1
4 2262-05-01 12:00:00  2262.2
5 3000-01-01 12:00:00  3000.0
>>> df.dtypes
ts       datetime64[us]
value           float64
dtype: object
>>>
>>> wr.s3.to_parquet(df, "s3://<REDACTED>/test_pandas_30/df.snappy.parquet")
{'paths': ['s3://<REDACTED>/test_pandas_30/df.snappy.parquet'], 'partitions_values': {}}
>>>
>>> result = wr.s3.read_parquet("s3://<REDACTED>/test_pandas_30/df.snappy.parquet")
>>> result
                             ts   value
0 2169-02-09 11:09:07.419103232  1000.0  # BAD: outside nanosecond range
1 2262-03-23 11:34:33.709551616  1677.1  # BAD: outside nanosecond range
2 1677-10-01 12:00:00.000000000  1677.2  # OK: inside nanosecond range
3 2262-04-01 12:00:00.000000000  2262.1  # OK: inside nanosecond range
4 1677-10-10 12:25:26.290448384  2262.2  # BAD: outside nanosecond range
5 1830-11-23 12:50:52.580896768  3000.0  # BAD: outside nanosecond range
>>> result.dtypes
ts       datetime64[ns]
value           float64
dtype: object

Expected behavior

The dtype should not change and data should be coming back as it was put in.

Your project

No response

Screenshots

No response

OS

Linux 6.6.114.1-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 20:46:23 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Python version

3.14.5

AWS SDK for pandas version

3.16.1

Additional context

$ uv pip list
Package           Version
----------------- -----------
awswrangler       3.16.1
boto3             1.43.24
botocore          1.43.24
jmespath          1.1.0
numpy             2.4.6
packaging         26.2
pandas            3.0.3
pyarrow           24.0.0
python-dateutil   2.9.0.post0
s3transfer        0.18.0
setuptools        82.0.1
six               1.17.0
typing-extensions 4.15.0
urllib3           2.7.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions