Add opt-in polars output via pandas pipeline + boundary adapter#2817
Add opt-in polars output via pandas pipeline + boundary adapter#2817dokson wants to merge 1 commit into
Conversation
|
@buzzvolt @edoaltamura Does this work for you? It still uses Pandas internally but optionally converts to Polars at very end. Much easier than porting all the internal logic to Polars. |
Closes ranaroussi#1868 (alternative to ranaroussi#2782 / ranaroussi#2808). Internal yfinance pipelines stay 100% pandas. Polars support is a thin output adapter applied at every public-API boundary: yf.config.dataframe.backend = 'polars' yf.Ticker('MSFT').history(period='1mo') # -> polars.DataFrame Direct answer to the maintenance concern raised on ranaroussi#2808: a parallel polars implementation doubles surface area; an output adapter does not. There is *one* code path for transformations. It isn't. yfinance frames are small enough that pandas vs polars makes no practical difference -- exactly the point ranaroussi#2808's review flagged. The value is *convenience*: users who downstream-process with polars get polars frames straight from yfinance instead of threading `pl.from_pandas(...)` after every call. - setup.py: optional dep group `yfinance[polars]` (polars only; no narwhals because with no transformation layer there is nothing to make backend-agnostic). - config.py: `YfConfig.dataframe.backend` defaults to 'pandas', raises ValueError on unknown values, leaves prior state intact on validation failure. - _backend.py: helpers (`current_backend`, `df_to_backend`, `series_to_backend`, `empty_df`) and public type aliases `DataFrameLike` / `SeriesLike`. All pure pass-through when backend == 'pandas'. Wrapped at the public boundary: - lookup.py, calendars.py, domain/{sector,industry,domain}.py - base.py: get_dividends/splits/capital_gains/actions/shares/ shares_full/earnings_dates/earnings, financial statements, recommendations, upgrades_downgrades, holders x6, valuation_measures, sustainability, analysis getters x6. - ticker.py: option_chain (calls / puts) and all @Property return annotations. - scrapers/funds.py: fund_operations, top_holdings, equity_holdings, bond_holdings. - scrapers/history.py: 4 return points of `history()`. - multi.py: `download()` final return. Type annotations: public-facing return types changed from `pd.DataFrame` / `pd.Series` to `DataFrameLike` / `SeriesLike` (union of pandas + polars). Not a breaking change: pandas remains a valid subtype, and `polars` is only imported at type-check time (TYPE_CHECKING guard). - Default backend is 'pandas'. Existing users see no behavioural difference. - `df_to_backend(df)` is identity (`assertIs`) when backend == 'pandas'. Verified by unit test. - Caches keep frames in pandas; conversion happens at each read, so changing backend mid-session is honoured immediately by every subsequent call. `tests/test_dataframe_backend.py` -- 37 tests, no network: - Config validation (default / accepted / unknown / no-corruption). - `df_to_backend` semantics (passthrough, polars conversion, named-index promotion, override, RangeIndex drop, empty frame). - `series_to_backend` semantics. - Lookup parity on both backends + empty result. - All 17 wrapped base.py getters return the right backend type, plus `as_dict=True` always returns a plain dict. - Backend switch invariant: pandas -> polars -> pandas in sequence returns the matching type each time. - Calendars `_to_backend` on both backends. - Sector.industries property switches with backend. All 58 unit tests (37 backend + 21 utils) pass. Ruff clean for the changed files. - `download()` long-form polars shape: kept as MultiIndex pandas converted via reset_index. Native polars long-form is a separate opinionated decision. - Price-repair engine stays pandas-internal (numerical kernel with scipy.ndimage, not idiomatic DataFrame ops). The pandas frame is converted at the same `history()` boundary.
58bfb62 to
5353ffa
Compare
|
That'd be a useful functionality, thanks! |
|
great @edoaltamura!!! I'm very happy that you appreciate this and that it can be useful to you |
One of the reasons why I chose to go with polars is that as I had put it in #2783 and docs/migration-v2-polars.md obviously for speed especially for group-by and to avoid things of multi-index. Instead of viewing it as a strict "either/or", many data teams are adopting hybrid workflows. You can do heavy data ingestion, filtering, and feature engineering in Polars, and then convert it into a Pandas object at the boundaries of your Machine Learning pipelines. |
Alternative to #2782 and #2808 — closes #1868.
Summary
Opt-in polars output without doubling the codebase. One line of user code:
Pandas remains the default. Polars is a single optional install:
pip install yfinance[polars].Why this differs from #2808
@ValueRaider's review on #2808 pointed out the core problem:
This PR takes that feedback as the design constraint. Instead of porting every scraper into a native polars implementation (#2808: ~5000 lines, including a 2272-line
polars_repair.py), the internal pipelines stay 100% pandas and polars support is a thin output adapter applied at every public-API boundary. There is one code path internally; polars never participates in transformations.Why this is not a performance play
It isn't. yfinance frames are small enough that pandas vs polars makes no practical difference — exactly @ValueRaider's point. The value here is convenience: users whose downstream pipelines are polars get polars frames straight from yfinance instead of threading
pl.from_pandas(...)after every call. That's it.No breaking changes
pandas. Existing users see no behavioural difference.df_to_backend(df)is identity (assertIs) when backend ==pandas. Unit-tested.pd.DataFrametoDataFrameLike = Union[pd.DataFrame, "pl.DataFrame"]. Pandas remains a valid subtype, so all existing pandas-typed callers continue to type-check. Polars is imported only underTYPE_CHECKING, so users withoutpolarsinstalled see no import error.Surface covered
Ticker.history()(4 return paths) andyf.download().Ticker.X@Property accessors (viabase.py get_X()wraps).Ticker.get_*methods returning DataFrame/Series: dividends, splits, capital_gains, actions, shares, shares_full, earnings_dates, earnings, financial statements, recommendations, upgrades_downgrades, holders ×6, valuation_measures, sustainability, analysis getters ×6.Ticker.option_chain()(calls + puts).Ticker.funds_data.{fund_operations, top_holdings, equity_holdings, bond_holdings}.Lookup.*,Calendars.*,Sector.industries,Industry.top_*,Domain.top_companies.Internal scraper
_quote.X / _holders.X / _analysis.X / _fundamentals.Xattributes are intentionally not wrapped — they are private (_-prefixed) and only used bybase.py, which wraps at the public boundary.Tests
tests/test_dataframe_backend.py— 37 tests, no network required:df_to_backendsemantics (passthrough on pandas, polars conversion, named-index promotion, override, RangeIndex drop, empty).series_to_backendsemantics.as_dict=Truealways returns a plain dict._to_backendon both backends.Ruff clean for every file touched.
Out of scope
download()long-form polars shape: kept as MultiIndex pandas converted via reset_index. Native polars long-form is a separate opinionated decision.scipy.ndimage, not idiomatic DataFrame ops. The pandas frame is converted at the samehistory()boundary.Test plan
python -m unittest tests.test_dataframe_backend tests.test_utils— 58 passruff checkclean on every changed file (the lone remaining E702 inhistory.py:2734is pre-existing and addressed by Simplify phantom-dividend repair branch + drive-by typo/lint fixes #2810)pyright— no new errors on the changed files