Commit 2b0e7d3
authored
Add ByteChannel API for raw byte streaming (#35)
* Add ByteChannel API for raw byte streaming
ByteChannel provides raw byte streaming between Erlang and Python without
term serialization overhead, suitable for HTTP bodies, file transfers, and
binary protocols.
Erlang API:
- py_byte_channel:new/0,1 - Create channel with optional backpressure
- py_byte_channel:send/2 - Send raw bytes
- py_byte_channel:recv/1,2 - Blocking receive with optional timeout
- py_byte_channel:try_receive/1 - Non-blocking receive
- py_byte_channel:close/1 - Close channel
Python API:
- ByteChannel class with send_bytes, receive_bytes, try_receive_bytes
- async_receive_bytes for asyncio compatibility
- Sync and async iteration support
Implementation reuses the existing py_channel_t infrastructure with new
NIF functions that skip term_to_binary/binary_to_term conversion.
* Simplify ByteChannel async_receive to match Channel pattern
Use simple polling (asyncio.sleep) like the existing Channel.async_receive()
instead of the more complex event loop dispatch integration. Both can be
upgraded to proper event-driven async in a future change.
Added async e2e test for ByteChannel.
* Add event-driven async receive for Channel and ByteChannel
Replaces polling with proper event loop integration:
- Register with channel via direct C method (no Erlang callback overhead)
- Wait for EVENT_TYPE_TIMER dispatch when data arrives
- Falls back to polling for non-Erlang event loops
New direct Python methods (bypass erlang.call):
- erlang._channel_wait(ch, callback_id, loop_capsule)
- erlang._channel_cancel_wait(ch, callback_id)
- erlang._byte_channel_wait(ch, callback_id, loop_capsule)
- erlang._byte_channel_cancel_wait(ch, callback_id)
When ErlangEventLoop is used:
1. async_receive registers handle in loop._timers
2. Direct C call registers waiter with channel
3. channel_send dispatches via event_loop_add_pending
4. Event loop _dispatch fires callback, resolves Future
5. No polling overhead
* Document event-driven async receive API
- Add behavior notes for async_receive() explaining event-driven vs polling
- Add event-driven async for ByteChannel documentation
- Add architecture diagram showing async receive flow with ErlangEventLoop
* Add benchmark comparing Channel, ByteChannel, and PyBuffer
Benchmark measures:
- Erlang send throughput (no Python)
- Erlang roundtrip (send + receive)
- Python receive performance
- Streaming throughput (1MB transfer)
Run with: escript examples/bench_byte_channel.erl
* Fix event-driven async: don't manipulate _handle_to_callback_id
Only use _timers dict for channel waiter registration. The
_handle_to_callback_id dict is for timer cancellation and shouldn't
be modified by channel waiters - this could cause race conditions
in free-threaded Python.1 parent fd8cef6 commit 2b0e7d3
File tree
13 files changed
+2204
-5
lines changed- c_src
- docs
- examples
- priv/_erlang_impl
- src
- test
13 files changed
+2204
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
37 | 51 | | |
38 | 52 | | |
39 | 53 | | |
| |||
0 commit comments