Skip to content

Stream Arrow RecordBatches using mpsc channel and tokio_streams#481

Merged
domoritz merged 1 commit intouwdata:dom/rust-streamfrom
jonmmease:jonmmease/stream_arrow_compiles
Aug 14, 2024
Merged

Stream Arrow RecordBatches using mpsc channel and tokio_streams#481
domoritz merged 1 commit intouwdata:dom/rust-streamfrom
jonmmease:jonmmease/stream_arrow_compiles

Conversation

@jonmmease
Copy link

Fixes compilation of #462 by using an mpsc channel and tokio_streams.

The reason this is more complicated that what you were trying @domoritz , is that the Arrow type provided by the duckdb crate is not Send. This means that it (and objects that depend on it) cannot be transferred across threads. This is why it doesn't work to build a stream using the stream! macro and return it from inside the conn.interact closure.

Using mpsc channels like I am here is a common workaround for dealing with non-send objects. Here the Arrow object stays in the thread that constructed it, and pushes the record batches through the channel back to the main thread. This channel is wrapped into a stream using the ReceiverStream from the tokio_stream crate.

I didn't actually test this, just got it compiling, so give it a try and see if it works the way you want.

@domoritz domoritz merged commit ffc034a into uwdata:dom/rust-stream Aug 14, 2024
@domoritz
Copy link
Member

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants