Skip to content

Commit 2b0e7d3

Browse files
authored
Add ByteChannel API for raw byte streaming (#35)
* Add ByteChannel API for raw byte streaming ByteChannel provides raw byte streaming between Erlang and Python without term serialization overhead, suitable for HTTP bodies, file transfers, and binary protocols. Erlang API: - py_byte_channel:new/0,1 - Create channel with optional backpressure - py_byte_channel:send/2 - Send raw bytes - py_byte_channel:recv/1,2 - Blocking receive with optional timeout - py_byte_channel:try_receive/1 - Non-blocking receive - py_byte_channel:close/1 - Close channel Python API: - ByteChannel class with send_bytes, receive_bytes, try_receive_bytes - async_receive_bytes for asyncio compatibility - Sync and async iteration support Implementation reuses the existing py_channel_t infrastructure with new NIF functions that skip term_to_binary/binary_to_term conversion. * Simplify ByteChannel async_receive to match Channel pattern Use simple polling (asyncio.sleep) like the existing Channel.async_receive() instead of the more complex event loop dispatch integration. Both can be upgraded to proper event-driven async in a future change. Added async e2e test for ByteChannel. * Add event-driven async receive for Channel and ByteChannel Replaces polling with proper event loop integration: - Register with channel via direct C method (no Erlang callback overhead) - Wait for EVENT_TYPE_TIMER dispatch when data arrives - Falls back to polling for non-Erlang event loops New direct Python methods (bypass erlang.call): - erlang._channel_wait(ch, callback_id, loop_capsule) - erlang._channel_cancel_wait(ch, callback_id) - erlang._byte_channel_wait(ch, callback_id, loop_capsule) - erlang._byte_channel_cancel_wait(ch, callback_id) When ErlangEventLoop is used: 1. async_receive registers handle in loop._timers 2. Direct C call registers waiter with channel 3. channel_send dispatches via event_loop_add_pending 4. Event loop _dispatch fires callback, resolves Future 5. No polling overhead * Document event-driven async receive API - Add behavior notes for async_receive() explaining event-driven vs polling - Add event-driven async for ByteChannel documentation - Add architecture diagram showing async receive flow with ErlangEventLoop * Add benchmark comparing Channel, ByteChannel, and PyBuffer Benchmark measures: - Erlang send throughput (no Python) - Erlang roundtrip (send + receive) - Python receive performance - Streaming throughput (1MB transfer) Run with: escript examples/bench_byte_channel.erl * Fix event-driven async: don't manipulate _handle_to_callback_id Only use _timers dict for channel waiter registration. The _handle_to_callback_id dict is for timer cancellation and shouldn't be modified by channel waiters - this could cause race conditions in free-threaded Python.
1 parent fd8cef6 commit 2b0e7d3

File tree

13 files changed

+2204
-5
lines changed

13 files changed

+2204
-5
lines changed

CHANGELOG.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,20 @@
3434

3535
### Added
3636

37+
- **ByteChannel API** - Raw byte streaming channel without term serialization
38+
- `py_byte_channel:new/0,1` - Create byte channel (with optional backpressure)
39+
- `py_byte_channel:send/2` - Send raw bytes to Python
40+
- `py_byte_channel:recv/1,2` - Blocking receive with optional timeout
41+
- `py_byte_channel:try_receive/1` - Non-blocking receive
42+
- Python `ByteChannel` class with:
43+
- `send_bytes(data)` - Send bytes back to Erlang
44+
- `receive_bytes()` - Blocking receive (GIL released)
45+
- `try_receive_bytes()` - Non-blocking receive
46+
- `async_receive_bytes()` - Asyncio-compatible async receive
47+
- Sync and async iteration (`for chunk in ch`, `async for chunk in ch`)
48+
- Reuses the same `py_channel_t` infrastructure but skips term encoding/decoding
49+
- Suitable for HTTP bodies, file streaming, and binary protocols
50+
3751
- **Automatic Env Reuse for Event Loop Tasks** - Functions defined via `py:exec(Ctx, Code)`
3852
can now be called directly using `py_event_loop:run/3,4`, `create_task/3,4`, and `spawn_task/3,4`
3953
without manual env passing. The process-local environment is automatically detected and used

0 commit comments

Comments
 (0)