Describe the bug
Hi,
pandas 3.0 is defaulting timestamps to microsecond resolution where possible by default:
https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#datetime-timedelta-resolution-inference
While it was possible in pandas 2.0 already to use timestamps that have resolutions other than nanoseconds, the default change now makes it a more prominent change.
When writing and reading parquet files with small and large timestamps, only supported by using microsecond resolution, the results come back and look like some overflow happened in-between.
Note that the pd.Timestamp.min (1677-09-21 00:12:43.145224193) and pd.Timestamp.max (2262-04-11 23:47:16.854775807) "boundaries" in pandas still exist and they have nanosecond resolution.
It seems that dates in that range are working fine, but dates outside are not.
Somewhere along the way, nanosecond resolution must be enforced.
How to Reproduce
# Python 3.14.5 (main, May 10 2026, 19:28:16) [Clang 22.1.3 ] on linux
# Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import awswrangler as wr
>>> df = pd.DataFrame({"ts": pd.to_datetime(["1000-01-01 12:00:00", "1677-09-01 12:00:00", "1677-10-01 12:00:00", "2262-04-01 12:00:00", "2262-05-01 12:00:00", "3000-01-01 12:00:00"]), "value": [1000, 1677.1, 1677.2, 2262.1, 2262.2, 3000]})
>>>
>>> df
ts value
0 1000-01-01 12:00:00 1000.0
1 1677-09-01 12:00:00 1677.1
2 1677-10-01 12:00:00 1677.2
3 2262-04-01 12:00:00 2262.1
4 2262-05-01 12:00:00 2262.2
5 3000-01-01 12:00:00 3000.0
>>> df.dtypes
ts datetime64[us]
value float64
dtype: object
>>>
>>> wr.s3.to_parquet(df, "s3://<REDACTED>/test_pandas_30/df.snappy.parquet")
{'paths': ['s3://<REDACTED>/test_pandas_30/df.snappy.parquet'], 'partitions_values': {}}
>>>
>>> result = wr.s3.read_parquet("s3://<REDACTED>/test_pandas_30/df.snappy.parquet")
>>> result
ts value
0 2169-02-09 11:09:07.419103232 1000.0 # BAD: outside nanosecond range
1 2262-03-23 11:34:33.709551616 1677.1 # BAD: outside nanosecond range
2 1677-10-01 12:00:00.000000000 1677.2 # OK: inside nanosecond range
3 2262-04-01 12:00:00.000000000 2262.1 # OK: inside nanosecond range
4 1677-10-10 12:25:26.290448384 2262.2 # BAD: outside nanosecond range
5 1830-11-23 12:50:52.580896768 3000.0 # BAD: outside nanosecond range
>>> result.dtypes
ts datetime64[ns]
value float64
dtype: object
Expected behavior
The dtype should not change and data should be coming back as it was put in.
Your project
No response
Screenshots
No response
OS
Linux 6.6.114.1-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 20:46:23 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Python version
3.14.5
AWS SDK for pandas version
3.16.1
Additional context
$ uv pip list
Package Version
----------------- -----------
awswrangler 3.16.1
boto3 1.43.24
botocore 1.43.24
jmespath 1.1.0
numpy 2.4.6
packaging 26.2
pandas 3.0.3
pyarrow 24.0.0
python-dateutil 2.9.0.post0
s3transfer 0.18.0
setuptools 82.0.1
six 1.17.0
typing-extensions 4.15.0
urllib3 2.7.0
Describe the bug
Hi,
pandas 3.0 is defaulting timestamps to microsecond resolution where possible by default:
https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#datetime-timedelta-resolution-inference
While it was possible in pandas 2.0 already to use timestamps that have resolutions other than nanoseconds, the default change now makes it a more prominent change.
When writing and reading parquet files with small and large timestamps, only supported by using microsecond resolution, the results come back and look like some overflow happened in-between.
Note that the
pd.Timestamp.min(1677-09-21 00:12:43.145224193) andpd.Timestamp.max(2262-04-11 23:47:16.854775807) "boundaries" in pandas still exist and they have nanosecond resolution.It seems that dates in that range are working fine, but dates outside are not.
Somewhere along the way, nanosecond resolution must be enforced.
How to Reproduce
Expected behavior
The
dtypeshould not change and data should be coming back as it was put in.Your project
No response
Screenshots
No response
OS
Linux 6.6.114.1-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 20:46:23 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Python version
3.14.5
AWS SDK for pandas version
3.16.1
Additional context