source-shopify 3.5.1: source container OOMs (160MiB → >2GiB in <8 min) at end of `products` stream BULK walk — distinct from #76969 sort fix

[geraldcor]

Connector Name

source-shopify

Connector Version

3.5.1

What step the error happened?

During the sync

Relevant information

Summary: The source container is OOMKilled by a fast, transient memory spike (~160MiB → >2GiB in under 8 minutes) during/immediately after the products stream's GraphQL BULK historical walk. Steady-state memory for the rest of the sync is ~100–160MiB. This is a different code path than the OOM fixed in 3.3.2 (#76969, disk-backed external sort in sort_output_asc()) — that fix is already included in 3.5.1 and our spike occurs during bulk record production, not post-read sorting.

Environment

• source-shopify 3.5.1 (airbyte/source-shopify:3.5.1)
• destination-bigquery 3.0.19, container-orchestrator 2.0.1
• Airbyte OSS, Helm chart v1, Kubernetes (DigitalOcean DOKS)
• Connection-level resource_requirements: memory_request 1Gi / memory_limit 2Gi (per container)

Sync shape

• 7 streams: customers, metafield_products, order_refunds, orders, products, +2 more
• Full historical sync (state reset after upgrading from the declarative-manifest connector), BULK slices of P30D walking from 2022 to present
• Most BULK jobs complete sub-second; slices are mostly small/empty (many checkpoints with recordCount=0)

Timeline of the failure (UTC)

18:26:26  sync running; BULK walk proceeding normally (metafield_products etc.)
18:26–18:45  streams customers, metafield_products, order_refunds, orders complete;
             products BULK walk proceeding. Source container memory sampled every
             20s via cgroup: 102–159MiB the entire time. CPU < 5% of a core.
18:45:02  products requests its FINAL slice:
          "Stream: `products` requesting BULK Job for period: 2026-06-28 ..."
          BULK Job CREATED → COMPLETED (sub-second)
18:45:03  last log line relayed by orchestrator (checkpoint flush). Total log
          silence from here on.
18:49     last cgroup sample before our sampler window ended: source at 158MiB
18:53:49  source container OOMKilled (exit 137) at the 2GiB cgroup limit

So the spike from ~160MiB to >2GiB happened in ≤8 minutes with no log output, right at/after the final products BULK slice completed.

Why we believe this is a distinct, untracked bug

• The JSONL bulk download/parse path in current master streams correctly (request_kwargs={"stream": True} + iter_content to disk in bulk/job.py; line-by-line for line in jsonl_file: in bulk/record.py) — so this is not whole-file buffering.
• fix(source-shopify): stream bulk GraphQL records through a disk-backed external sort to prevent OOM #76969 (shipped 3.3.2) fixed the in-memory sorted() over the full emitted-record iterable; that fix is present in 3.5.1 and our profile doesn't match it (that one grew during record emission with logs flowing; ours spikes silently at a slice boundary).
• A remaining candidate from reading bulk/record.py: the per-parent record-components buffer (top-level record + all __parentId-linked children held until the next top-level record boundary). products is the most deeply nested bulk query (variants, images, media, metafields), so a query shape/ordering edge case that defeats the new-record boundary detection would produce exactly this unbounded growth. We have not confirmed this hypothesis — offered as a starting point.

Workaround: raising connection-level memory_limit to 4Gi lets the sync proceed (memory_request unchanged). Note we had to apply this via the internal config API / connection-level resource_requirements because of the known Helm v1 regression where JOB_MAIN_CONTAINER_* env vars are ignored for replication pods (#68162, #30814).

Relevant log output

2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867094621` time elapsed: ...
2026-07-08 18:45:02,913 ... Stream: `products` requesting BULK Job for period: 2026-06-28T00:00:00+00:00 -- ...
2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867127389` is CREATED
2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867127389` is COMPLETED
2026-07-08 18:45:03,018 ... CheckpointManager(flushStreamCheckpoint) [last line before silence]

(no further output; source container OOMKilled exit 137 at 18:53:49)

Kubernetes container status at failure:

source: reason=OOMKilled exit=137 finished=2026-07-08T18:53:49Z
(orchestrator and destination containers unaffected)

Sampled source-container memory (cgroup memory.current, 20s interval, 18:26–18:49): min 102MiB, max 159MiB — the entire spike to >2GiB occurred in the final ~5–8 minutes with no log output.

---

Internal Tracking: https://github.com/airbytehq/oncall/issues/13076

[octavia-bot]

AI Triage Starting...

Thank you for opening this issue! I'm an AI assistant that will do an initial review and may:

• Ask clarifying questions if needed
• Suggest relevant documentation or workarounds
• Flag this for internal visibility

Note: Community issues are reviewed on a best-effort basis and do not have a guaranteed response time. For additional help:

• Join the Airbyte Community Slack to connect with other users and community members
• If you're an Airbyte Cloud customer, you can open a support ticket or contact your account representative, referencing this issue URL

View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/a6ecb2cf924b4463af460fc32279b1a8

✅ Devin AI session created successfully!

[airbyte-support-bot]

Hi Greg Corey (@geraldcor), thank you for the exceptionally detailed report — the memory samples, timeline, and code-path analysis are very helpful.

We're looking into this and have escalated it to our engineering team for investigation. We haven't drawn any conclusions yet, but your notes on the products BULK walk and the per-parent record-components buffer in bulk/record.py give them a solid starting point.

In the meantime, the workaround you found (raising the connection-level memory_limit) is a reasonable mitigation while this is investigated.

We've escalated this for investigation and are tracking it internally. If anything else surfaces (e.g. it reproduces on a narrower date range or a specific product shape), feel free to add it here.

Please ensure you mask or remove any sensitive information (API keys, passwords, tokens, connection strings) before sharing any additional logs.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.