Thoughts on my Implementation

[Biztactix-Ryan]

G'day,
I already use your FullTextIndex for a couple of things, really love it,
What I'm looking to do is create a replacement for PST files essentially, a way to store long term Emails and Index them for retrieval.
However I don't want to use a segments folder like you've got, I'd like it to a be a single large file,
So I've created IO Classes for Zonetree and already have it working, It's not quite done as there's still some other non Zonetree things having issues.

But essentially I took the concept of how REFS works by storing Extents(Segments) as blocks in a single file, Each time a new Segment is written we essentially write it to the end of the file, unless it's been recently written then we overwrite the existing one,
By storing mulitple copies of our Segment and our various other structures, it gives us alot o redundancy for recovery, we also can cleanup by either using our stored metadata to copy out the segments we need, OR to walk the whole file and grab the latest version of each segment.

I wanted to get your opinion on the overall concept? , I know it's alot of redundancy, but I'm happy to trade storage for resiliancy.
By keeping track of the various segments, It's pretty fast to return segments as zone tree needs, I haven't done a full benchmark on it yet.
But was just wondering if you had any ideas to improve it,
ATM, I'm handling the RandomAccessDeviceProvider, FileStreamProvider and the WriteAheadLogProvider, and it seems to work

[koculu]

Hi @Biztactix-Ryan

Writing a filesystem from scratch can be tricky, especially without prior experience. I haven’t gone down that path myself, but I appreciate your goal. There’s a ton of info available these days—thanks to the power of LLMs—that might help you out.

I did a brief look into ReFS, and it does sound promising. It would be awesome if you create an open-source library that others (including ZoneTree) can use. For my own workflows, I’m comfortable with multiple files, but I’d love to see how your single-file approach evolves. Please keep us posted!

[Biztactix-Ryan]

I'm currently digging into the FullText Search code where I see you create a new Zonetree database of CompositeKeyOfTokenRecordPrevious<Trecord.TToken>
I'm not seeing any good way to insert our RandomAccessDeviceManager and WriteAheadLogProvider into them
As HashedSearchEngine creates an IndexOfTokenRecordPreviousToken which creates the Factory where we can inject our DataType
However It also creates
SearchOnIndexOfTokenRecordPreviousToken
AdvancedSearchOnIndexOfTokenRecordPreviousToken

So from what I see, I need to Wrap

• HashedSearchEngine
• IndexOfTokenRecordPreviousToken
• SearchOnIndexOfTokenRecordPreviousToken
• AdvancedSearchOnIndexOfTokenRecordPreviousToken

As they are all Concretes and sealed which point to each other, And that's as far as I've dug, It'll likely be a few more...
Am I right? or have I missed something pretty obvious?

What I'd say would make it easier, is an overload constructor to pass down a ZoneTree Factory into HashedSearchEngine
So that way if you have any custom setup for the Zonetree Factory you can pass it down.

I could use the new Method Overwriting features in C# to just overwrite the methods I need, But I Thought I'd check before I go duplicating all this code.

[koculu]

Please check the new released version that supports advanced customizations.
https://github.com/koculu/ZoneTree.FullTextSearch/releases/tag/release-v1.0.8

PS: It would be best if we discuss ZoneTree.FullTextSearch related issues in its repository but not here.

[Biztactix-Ryan]

Mate!!!
Did you just fix all my problems in a single go?

Also happy to discuss over there, I just thought since it was referencing what I'd already posted...

But Seriously... That's awesome... I was just about to finish up for the day on this stuff.. Might stick at it for a while longer now!

[koculu]

Not a big deal.
It was a simple change. Adding a couple of parameters to the constructors. 😄

[Biztactix-Ryan]

Saves me plenty of work, which I was going to put off for a few days