Skip to content

Add opt-in polars output via pandas pipeline + boundary adapter#2817

Open
dokson wants to merge 1 commit into
ranaroussi:devfrom
dokson:feature/optional-polars-output
Open

Add opt-in polars output via pandas pipeline + boundary adapter#2817
dokson wants to merge 1 commit into
ranaroussi:devfrom
dokson:feature/optional-polars-output

Conversation

@dokson

@dokson dokson commented May 15, 2026

Copy link
Copy Markdown
Contributor

Alternative to #2782 and #2808closes #1868.

Summary

Opt-in polars output without doubling the codebase. One line of user code:

yf.config.dataframe.backend = 'polars'
yf.Ticker('MSFT').history(period='1mo')   # → polars.DataFrame

Pandas remains the default. Polars is a single optional install: pip install yfinance[polars].

Why this differs from #2808

@ValueRaider's review on #2808 pointed out the core problem:

The main problem with this, is now we have to maintain Polars for no performance improvement — yfinance tables aren't big enough for choice to matter.

This PR takes that feedback as the design constraint. Instead of porting every scraper into a native polars implementation (#2808: ~5000 lines, including a 2272-line polars_repair.py), the internal pipelines stay 100% pandas and polars support is a thin output adapter applied at every public-API boundary. There is one code path internally; polars never participates in transformations.

This PR #2808
LoC added ~700 ~5000
Polars-native repair engine none (pandas + convert) 2272 lines
Polars-native scrapers none every scraper
Default backend pandas (no break) pandas (no break)
Polars dependency extras_require extras_require
Performance unchanged unchanged
Maintenance surface +1 adapter file, ~700 lines duplicate every transform

Why this is not a performance play

It isn't. yfinance frames are small enough that pandas vs polars makes no practical difference — exactly @ValueRaider's point. The value here is convenience: users whose downstream pipelines are polars get polars frames straight from yfinance instead of threading pl.from_pandas(...) after every call. That's it.

No breaking changes

  • Default backend is pandas. Existing users see no behavioural difference.
  • df_to_backend(df) is identity (assertIs) when backend == pandas. Unit-tested.
  • Public-facing return annotations are widened from pd.DataFrame to DataFrameLike = Union[pd.DataFrame, "pl.DataFrame"]. Pandas remains a valid subtype, so all existing pandas-typed callers continue to type-check. Polars is imported only under TYPE_CHECKING, so users without polars installed see no import error.
  • Caches keep frames in pandas; conversion happens on each public read, so switching backend mid-session is honoured immediately by every subsequent call.

Surface covered

  • Ticker.history() (4 return paths) and yf.download().
  • All 40+ Ticker.X @Property accessors (via base.py get_X() wraps).
  • All Ticker.get_* methods returning DataFrame/Series: dividends, splits, capital_gains, actions, shares, shares_full, earnings_dates, earnings, financial statements, recommendations, upgrades_downgrades, holders ×6, valuation_measures, sustainability, analysis getters ×6.
  • Ticker.option_chain() (calls + puts).
  • Ticker.funds_data.{fund_operations, top_holdings, equity_holdings, bond_holdings}.
  • Lookup.*, Calendars.*, Sector.industries, Industry.top_*, Domain.top_companies.

Internal scraper _quote.X / _holders.X / _analysis.X / _fundamentals.X attributes are intentionally not wrapped — they are private (_-prefixed) and only used by base.py, which wraps at the public boundary.

Tests

tests/test_dataframe_backend.py — 37 tests, no network required:

  • Config validation (default / accepted / unknown / no-corruption-on-failure).
  • df_to_backend semantics (passthrough on pandas, polars conversion, named-index promotion, override, RangeIndex drop, empty).
  • series_to_backend semantics.
  • Lookup parity on both backends + empty result.
  • All 17 wrapped base.py getters return the right backend type, plus as_dict=True always returns a plain dict.
  • Backend switch invariant: pandas → polars → pandas in sequence returns the matching type each time.
  • Calendars _to_backend on both backends.
  • Sector.industries property switches with backend.
$ python -m unittest tests.test_dataframe_backend tests.test_utils
Ran 58 tests in 0.290s
OK (expected failures=1)

Ruff clean for every file touched.

Out of scope

  • download() long-form polars shape: kept as MultiIndex pandas converted via reset_index. Native polars long-form is a separate opinionated decision.
  • Price-repair engine stays pandas-internal — it's a numerical kernel with scipy.ndimage, not idiomatic DataFrame ops. The pandas frame is converted at the same history() boundary.

Test plan

@ValueRaider

ValueRaider commented May 18, 2026

Copy link
Copy Markdown
Collaborator

@buzzvolt @edoaltamura Does this work for you? It still uses Pandas internally but optionally converts to Polars at very end. Much easier than porting all the internal logic to Polars.

Closes ranaroussi#1868 (alternative to ranaroussi#2782 / ranaroussi#2808).

Internal yfinance pipelines stay 100% pandas. Polars support is a
thin output adapter applied at every public-API boundary:

    yf.config.dataframe.backend = 'polars'
    yf.Ticker('MSFT').history(period='1mo')   # -> polars.DataFrame

Direct answer to the maintenance concern raised on ranaroussi#2808: a parallel
polars implementation doubles surface area; an output adapter does
not. There is *one* code path for transformations.

It isn't. yfinance frames are small enough that pandas vs polars
makes no practical difference -- exactly the point ranaroussi#2808's review
flagged. The value is *convenience*: users who downstream-process
with polars get polars frames straight from yfinance instead of
threading `pl.from_pandas(...)` after every call.

- setup.py: optional dep group `yfinance[polars]` (polars only;
  no narwhals because with no transformation layer there is
  nothing to make backend-agnostic).
- config.py: `YfConfig.dataframe.backend` defaults to 'pandas',
  raises ValueError on unknown values, leaves prior state intact
  on validation failure.
- _backend.py: helpers (`current_backend`, `df_to_backend`,
  `series_to_backend`, `empty_df`) and public type aliases
  `DataFrameLike` / `SeriesLike`. All pure pass-through when
  backend == 'pandas'.

Wrapped at the public boundary:

- lookup.py, calendars.py, domain/{sector,industry,domain}.py
- base.py: get_dividends/splits/capital_gains/actions/shares/
  shares_full/earnings_dates/earnings, financial statements,
  recommendations, upgrades_downgrades, holders x6,
  valuation_measures, sustainability, analysis getters x6.
- ticker.py: option_chain (calls / puts) and all @Property
  return annotations.
- scrapers/funds.py: fund_operations, top_holdings,
  equity_holdings, bond_holdings.
- scrapers/history.py: 4 return points of `history()`.
- multi.py: `download()` final return.

Type annotations: public-facing return types changed from
`pd.DataFrame` / `pd.Series` to `DataFrameLike` / `SeriesLike`
(union of pandas + polars). Not a breaking change: pandas remains
a valid subtype, and `polars` is only imported at type-check time
(TYPE_CHECKING guard).

- Default backend is 'pandas'. Existing users see no behavioural
  difference.
- `df_to_backend(df)` is identity (`assertIs`) when backend ==
  'pandas'. Verified by unit test.
- Caches keep frames in pandas; conversion happens at each
  read, so changing backend mid-session is honoured immediately
  by every subsequent call.

`tests/test_dataframe_backend.py` -- 37 tests, no network:

- Config validation (default / accepted / unknown / no-corruption).
- `df_to_backend` semantics (passthrough, polars conversion,
  named-index promotion, override, RangeIndex drop, empty frame).
- `series_to_backend` semantics.
- Lookup parity on both backends + empty result.
- All 17 wrapped base.py getters return the right backend type,
  plus `as_dict=True` always returns a plain dict.
- Backend switch invariant: pandas -> polars -> pandas in sequence
  returns the matching type each time.
- Calendars `_to_backend` on both backends.
- Sector.industries property switches with backend.

All 58 unit tests (37 backend + 21 utils) pass. Ruff clean for
the changed files.

- `download()` long-form polars shape: kept as MultiIndex pandas
  converted via reset_index. Native polars long-form is a
  separate opinionated decision.
- Price-repair engine stays pandas-internal (numerical kernel
  with scipy.ndimage, not idiomatic DataFrame ops). The pandas
  frame is converted at the same `history()` boundary.
@dokson dokson force-pushed the feature/optional-polars-output branch from 58bfb62 to 5353ffa Compare May 18, 2026 21:02
@edoaltamura

Copy link
Copy Markdown

That'd be a useful functionality, thanks!

@dokson

dokson commented May 19, 2026

Copy link
Copy Markdown
Contributor Author

great @edoaltamura!!!

I'm very happy that you appreciate this and that it can be useful to you

@buzzvolt

buzzvolt commented May 20, 2026

Copy link
Copy Markdown

@buzzvolt @edoaltamura Does this work for you? It still uses Pandas internally but optionally converts to Polars at very end. Much easier than porting all the internal logic to Polars.

One of the reasons why I chose to go with polars is that as I had put it in #2783 and docs/migration-v2-polars.md obviously for speed especially for group-by and to avoid things of multi-index.

Instead of viewing it as a strict "either/or", many data teams are adopting hybrid workflows. You can do heavy data ingestion, filtering, and feature engineering in Polars, and then convert it into a Pandas object at the boundaries of your Machine Learning pipelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants