TrailBase sync

[ignatz]

First and foremost wanted to say "hi" - Hi!

I have a project https://github.com/trailbaseio/trailbase, which is a SQLite-based potpourri of backend/database functionality (think Firebase, Supabase, PocketBase, ...). Among other things, you can subscribe to data changes, which I figured could be a good foundation for an efficient TanStack/db integration.

I did a minimal PoC: https://github.com/trailbaseio/trailbase/blob/main/examples/local-first/src/lib/trailbase.ts#L36, which seems to work (there's a good chance I'm still holding things wrong). Loving TanStack/db, I though this could turn out pretty sweet.

While still being early, I wanted to reach out hoping to hear your thoughts and maybe establishing an open channel or even collaboration. Either way, thanks for TanStack/db ❤️

[KyleAMathews]

Hey! It's very cool seeing your implementation! Looks very simple — which is the plan/hope :-D

We'd love to have you onboard helping collaborate. We want to build an incredibly good client lib for any sort of data loading strategy so having your unique insights, etc. around would be great.

Would you like to PR your trailbase implementation to https://github.com/TanStack/db/tree/main/packages/db-collections?

[KyleAMathews]

couldn't electric just return the actual id or even the effectively inserted record? Presumably, that would simplify electric's integration (and even improve latency)

Electric doesn't do the writes. So it's the user's responsibility to write and return the txid.

[KyleAMathews]

another random question: Is there an approach for persistence right now? Specifically, I would like to load the last known state from local storage while the initial fetch is catching us up. Once the initial fetch completes the stale state should be replaced entirely. Something like that or similar would be useful to support offline use-cases.

There's not yet built in. We're not sure if persistence like this will be a framework feature or a responsibility of the sync too as it can vary what's the best way to do persistentance e.g. electric leverages the http cache.

See more here #82

[ignatz]

Continuing this thread, would you still be open to upstreaming the TrailBase integration?

No worries, if not. If so, what would be the next steps? How can I best support you, e.g. adding a test harness etc? Are there other ways I could help you out? Thanks

[KyleAMathews]

For sure! Next step is just a PR w/ your options creator + tests similar to what's there for electric/query & then an example. The current example is for a pg/api backend so probably you should just create a separate example that shows off how TrailBase works w/ DB.

[ignatz]

Closing the loop, TanStack folks just published the TrailBase integration: https://www.npmjs.com/package/@tanstack/trailbase-db-collection.

Thank you so much Kyle, I had a blast working with you. Looking forward to collaborating more going forward. 🥳 🙏

❤️ TanStack/db

[KyleAMathews]

Just updated the TrailBase integration to mimic electric's for now: https://github.com/trailbaseio/trailbase/blob/dev/examples/local-first/src/lib/trailbase.ts#L44 .

Very cool to see built-in mutations here! Curious if you have a way for custom mutators on the backend? If so, are those exposed? You perhaps could auto generate actions for use in the frontend with createOptimisticAction #210

[ignatz]

Curious if you have a way for custom mutators on the backend?

I'm not sure if you're referring to something very specific? You can implement your own actions/endpoints that can do arbitrary operations. There are default crate/update/delete actions for tables. You can also generate strongly typed data bindings from the database schema.

You perhaps could auto generate actions for use in the frontend with createOptimisticAction #210

Could you elaborate a bit? Thanks!

[KyleAMathews]

A client DB action pairs an optimistic mutation with a server mutation. So for your implementation you've done that nicely for simple inserts/updates/deletes but people will also do more complex mutations. These don't need to be (and perhaps shouldn't be) auto wrapped up but depending on your architecture it might be possible.

So anything that the client can't predict for sure what the DB change should be e.g. atomic increments

[ignatz]

Thanks for explaining. Given the simple CRUD semantics, the client can make fairly accurate predictions (except maybe for computed values (values, atomic increments, ... as you mention) and failures such as validation, collisions, ...).

You perhaps could auto generate actions for use in the frontend with createOptimisticAction #210

I guess I most just don't understand what that would look like. Based on our prior conversation on the lifecycle, my understanding is that collection.insert({...}) as is, already performs an optimistic action, i.e. it creates an overlay with the insert and drops it when onInsert completes (and likewise for delete/update).

When you say "you perhaps could auto generate actions for use in the frontend with createOptimisticAction", is there anything more that could be done for insert/update/delete or are you thinking of a more general action concept (e.g. a composition of mutations)?

[KyleAMathews]

More general action composition. onInsert, onUpdate, onDelete respond to single direct collection mutations where you're directly mutating state.

createOptimisticAction or createTransaction are for composing less direct/simple mutations e.g. you're mutating multiple collections in a transaction or the client-side mutations get turned into one API call, etc.

[ignatz]

Got it - thanks! User-defined action are free-form JS/TS, i.e. I can't think of anything that could be auto-generated at the moment.