Can you do this, but async?

[nichoth]

I was thinking about a situation of doing a 'live' map-reduce operation — where you group multiple log values under a key (this is the map function, choosing the key), and then reducing to get the value for that key. In order to get the previous reduced value from the leveldb though, you would need to read the current value for this key, which is an async operation.

I have a feeling that it could get weird when you try to keep all the constraints we want, like atomic writes, and also optimizing the writes by batching a number of input values. I haven't thought this through too much yet, but figured it wouldn't hurt to think out loud.

[dominictarr]

I have day dreamed about this occasionally. the naive way is to just get the current value, then write the new value before processing the next one. easy because of pull streams, but probably slow (how slow?)

You could make that much faster by adding caching (caching is basically go-fast-sauce, just squirt it on your database and it goes faster) but also adds edge cases... like, don't evict something from the cache if it's being written. if you update a key, but it was already being written, probably better wait for it to be written before you write it again, incase the second write lands first.

hmm, probably a simple way would be to do a bunch of writes as a batch, on all dirty records.

[dominictarr]

the most important thing, is that the state of the view should be consistent with some sequence number. you shouldn't just write a single record at a time, you need to write them in order, or rather, in a batch, up to some point.

[mixmix]

I'm interested in doing async map's as well.

To do this it looks like this would be a major version change... changing the signature of map, and inside here probably changing the signature of pull-write as well to have a async reduce ? hmm that might be hard / make unpredicatable queues?

[dominictarr]

@mixmix when designing a thing like a database, it's important to think about what you are actually trying to do. What is an example of a query you are wanting to write?