Skip to content

perf(api_core): avoid full-corpus scans in list_funcs/func_query/entity_query pagination - #484

Open
lich0821 wants to merge 2 commits into
mrexodia:mainfrom
lich0821:perf/entity-query-pagination
Open

perf(api_core): avoid full-corpus scans in list_funcs/func_query/entity_query pagination#484
lich0821 wants to merge 2 commits into
mrexodia:mainfrom
lich0821:perf/entity-query-pagination

Conversation

@lich0821

Copy link
Copy Markdown

Problem

list_funcs, func_query, and entity_query all built the entire
function/global/import corpus into a Python list before filtering and
paginating it — even for a bounded page (e.g. count=50). On binaries with
hundreds of thousands of functions this dominates per-call latency, and
_collect_entities("functions", ...) always paid for a per-row segment-name
lookup and a type-info probe even when the caller's requested fields never
use them.

Fix

  • Add lazy_paginate_filter (utils.py): walks the underlying iterator
    lazily and stops as soon as enough matching rows are collected for the
    requested page, instead of materializing every row up front. Falls back to
    full materialization only when count=0 (unbounded page), since computing
    next_offset correctly still requires the full filtered set in that case.
  • _collect_entities takes needs_segment/needs_has_type flags so
    entity_query can skip the segment-name lookup and type-info probe per
    function when neither the active filter nor the requested output fields
    need them.
  • list_funcs/func_query switched to the lazy path for bounded pages.

Existing callers and output shape are unchanged; this only changes how many
rows get built before filtering stops.

Test plan

  • Existing test_api_core.py pagination/filter assertions pass unchanged
    (ran via ida-mcp-test tests/crackme03.elf / typed_fixture.elf)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant