[QDP] User Python API interface features and refactoring — Roadmap

[rich7420]

Summary

This roadmap defines the implementation path for QDP pipeline tuning (PR3): instrumenting the pipeline with observability, collecting baselines, introducing hardware-aware chunk and pool sizing, and replacing periodic sync with event-based buffer reuse—so that throughput and host-to-device overlap improve in a measurable, data-driven way.

Phase 1: [QDP] Add observability tools to diagnose pipeline performance #945

Phase 2: Baseline and Data Collection #972

Capture reproducible pre-optimization baselines for later comparison.

Phase 3: Add a Quantum Data Loader and API refactor #1000

Phase 4: Add Iterator for File Data Source in Python API #1010

Phase 5: refactor Reader and add ChunkedReader

Remove periodic sync_copy_stream() and reuse pinned buffers only after per-slot copy-complete events.

Phase Order and Dependencies

• Phase 2 depends only on Phase 1; run first and publish baseline.
• Phase 3 can be developed in parallel with Phase 4; recommend merging C3 before C4 so C4 uses configurable pool/chunk.
• Phase 4 should be validated by re-running Phase 2 and comparing overlap, throughput, and latency.

Alternatives Considered

Have you thought of other ways to solve the same problem?

Additional Context

Any other relevant information or resources.

[400Ping]

I am interested in phase2 & 3.

[rich7420]

Sorry, I’m working on it. But if you could review those PRs, I’d be grateful.