BUG: IndexError in screener.base._get_table when Finviz returns duplicate column headers

[Pirat83]

finvizfinance version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the master branch of finvizfinance.

Reproducible Example

import sys
from finvizfinance.screener.custom import Custom
from finvizfinance.screener import get_custom_screener_columns

c = Custom()
c.set_filter(filters_dict={"Country": "USA"})
c.screener_view(
    limit=sys.maxsize,
    columns=list(get_custom_screener_columns()),
)

Traceback:

File ".../finvizfinance/screener/base.py", line 127, in _get_table
    info_dict[table_header[i]] = col.text
              ~~~~~~~~~~~~^^^
IndexError: list index out of range

The bug also reproduces on 1.2.0 and master (the affected
_get_table / _parse_table code is byte-identical across those versions
and was last touched on 2024-09-16). It does not require sys.maxsize —
any limit that crosses the 20-row page boundary triggers it.

Issue Description

Base._get_table builds each parsed row as a dict keyed by header name.
When the Finviz response contains two <th> cells with the same text
(currently observed: column id 75 Dividend is emitted alongside the
separate Dividend Yield / Payout columns, so Dividend appears twice in
the header), the dict silently collapses both keys into one. Page 1's
resulting DataFrame therefore has one fewer column than the actual
header list (85 cols vs 86 <th>/<td> cells per row).

On the next page, _parse_table re-derives table_header from
df.columns:

table_headers = list(df.columns)  # length 85 after the dict collapse

The page-2 cell loop in _get_table then indexes table_header by cell
position. Each row has 86 cells, so when i == 85 the lookup
table_header[i] overflows and IndexError: list index out of range
fires.

Empirically, requesting any column subset whose len <= 73 works (no
duplicates surface). Including all 87 column ids triggers the duplicate
Dividend header and the failure.

Expected Behavior

screener_view(...) returns a multi-page DataFrame containing one row
per ticker, with both Dividend columns preserved (pandas allows
duplicate column labels). For the repro filter at the time of writing
this is 64 rows × 86 columns.

Installed Versions

Details

finvizfinance.__version__ == '1.3.0'  (also reproduced on 1.2.0 and master)
Python 3.11.15

Proposed fix

A patch is in #150: build each row as a positional list of values and
assemble via pd.DataFrame(frame, columns=table_header). Pandas
preserves duplicate column labels, so both Dividend columns survive
and the cell-vs-header indexing stays aligned across pages. The PR also
adds defensive bounds (skip surplus cells, pad missing cells with
None) so future Finviz HTML drift fails soft, and three regression
tests in test/test_screener.py using synthetic BeautifulSoup so they
don't depend on live Finviz markup.