[RFC] Support pluggable and composable per-field codecs

[reta]

History

A while ago we introduced CodecServiceFactory to provide an ability for the plugins to supply own custom codecs. It does the job and is actively used by:

• custom-codecs
• k-nn
• neural-search
• security-analytics
• jvector

Is your feature request related to a problem? Please describe

The plugins from above supply own codecs using CodecServiceFactory. One of the major limitations that CodecServiceFactory has is composability - for particular index: only one CodecServiceFactory could be provided, otherwise the exception is going to be raised (see please https://forum.opensearch.org/t/knn-doesnt-coexist-with-seismic-sparse/27709 as one of the recent examples).

Currently, there are two major uses of CodecServiceFactory, mutually exclusive in the scope of the same index:

1. Register additional codec with non-trivial instantiation (custom-codecs fe)
2. Wrap every single codec (k-nn, neural-search, security-analytics, jvector)

The first issue is already addressed by #20411, the second one needs more upfront design and decisions, hence this proposal.

Where We Are

The majority of the plugins (k-nn, neural-search, security-analytics, jvector) follow the same pattern of wrapping whatever codec index uses, for example:

public class KNNCodecService extends CodecService {
    ...
    @Override
    public Codec codec(String name) {
        return KNN1030Codec.builder()
            .delegate(super.codec(name))
            .mapperService(mapperService)
            .knnVectorsFormat(new KNN9120PerFieldKnnVectorsFormat(Optional.ofNullable(mapperService), nativeIndexBuildStrategyFactory))
            .build();
    }
}

Or

public class SparseCodecService extends CodecService {
    ...
    @Override
    public Codec codec(String name) {
        return new SparseCodec(super.codec(name));
    }
}

It somewhat works when index is referencing only one plugin (fe k-nn), but falls apart if index whats to benefit from several plugins (neural-search, k-nn), see please https://forum.opensearch.org/t/knn-doesnt-coexist-with-seismic-sparse/27709.

In fact, what most of the plugins need is just to encode / decode one single field (knn_vector, or sparse_vector_field), and that could be done in the scope of the same index (since every field has only one type, one of the exceptions here are stored fields which we will touch upon later). It is not possible at the moment but we believe it could be solved.

Describe the solution you'd like

There are multiple ways we could approach the problem.

Option 1. Support deep nested wrappings (fe new SparseCodec(new KNN1030Codec(super.codec(name)))). At high level, it looks like an easy fix but:
a) since there is no way to control plugins initialization sequence / ordering, it is very fragile and difficult to fix, and
b) there could be conflicts when different plugins alter the same stored fields (like _source fe)

Option 2. Introduce per-field customizations (see please #20411 (comment) for the context) that would allow to reliably collect per field PostingFormat / KnnVectorFormat / DocValuesFormat, and per-index StoredFieldsFormat (still backed by per-field customizations that would help in detecting conflicts).

The 2nd option is clearly superior although it requires large scale changes (that, luckily, we could rollout incrementally). To reduce the scope of the changes but keep the door fully open for any future enhancements, we are going to:

• focus on KnnVectorFormat / PostingsFormat, and
• focus on binary fields only (since this is how vectors are stored now)
• synthetic per-field StoredFieldsFormat will not be supported

This is what k-nn, neural-search, security-analytics, jvector plugins do override - there are no limitations in supporting all features but only efforts and amount of additional APIs.

Initial Design

Apache Lucene has pretty good out-of-the-box support of per-field abstraction (PerFieldKnnVectorsFormat, PerFieldPostingFormat, ...) which we could leverage (and do already in different places). What we need it to allow plugins to signal that they have their own formats and allow those to be injected into the default index codec. For this reasons, we may introduce the new PerFieldCodecBuilder abstraction:

import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
import org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat;
import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

public interface PerFieldCodecBuilder {
    Optional<PerFieldDocValuesFormat> docValuesFormat();
    Optional<PerFieldKnnVectorsFormat> knnVectorsFormat();
    Optional<PerFieldPostingsFormat> postingsFormat();
}

And new extension points for it in Plugin APIs:

Optional<PerFieldCodecBuilder> getPerFieldCodecBuilder(@Nullable MapperService mapperService, IndexSettings indexSettings);

The APIs purposedly use Optional and not Collection since all PerFieldXxx expose abstractions getXxxForField methods, naturally supporting multiple fields. It is also doubtful that individual plugin needs multiple PerFieldCodecBuilder. The DelegatingPerFieldCodecBuilder would collect the PerFieldCodecBuilder intances from all plugins:

public class DelegatingPerFieldCodecBuilder {
    private final Collection<PerFieldCodecBuilder> builders;

    public DelegatingPerFieldCodecBuilder(Collection<PerFieldCodecBuilder> builders) {
        this.builders = builders;
    }

    public DocValuesFormat buildDocValues(DocValuesFormat delegate) {
        return new CompositeDelegatingPerFieldDocValues(
            delegate,
            builders.stream().map(PerFieldCodecBuilder::docValuesFormat)
        );
    }

    public KnnVectorsFormat buildKnnVectors(KnnVectorsFormat delegate) {
        return new CompositeDelegatingPerFieldKnnVectors(
            delegate,
            builders.stream().map(PerFieldCodecBuilder::knnVectorsFormat)
        );
    }

    public PostingsFormat postingsFormat(PostingsFormat delegate) {
        return new CompositeDelegatingPerFieldPostingsFormat(
            delegate,
            builders.stream().map(PerFieldCodecBuilder::postingsFormat)
        );
    }

}

And used by CompositeDelegatingPerFieldCodec later on:

public class CompositeDelegatingPerFieldCodec extends FilterCodec {
    private static final String NAME = "CompositeDelegating103PerFieldCodec";
    private final DelegatingPerFieldCodecBuilder builder;

    public CompositeDelegatingPerFieldCodec(Codec delegate, DelegatingPerFieldCodecBuilder builder) {
        super(NAME, delegate);
        this.builder = builder;
    }

    @Override
    public DocValuesFormat docValuesFormat() {
        return builder.buildDocValues(delegate.docValuesFormat());
    }

    @Override
    public KnnVectorsFormat knnVectorsFormat() {
        return builder.buildKnnVectors(delegate.knnVectorsFormat());
    }

    @Override
    public PostingsFormat postingsFormat() {
      return builder.postingsFormat(delegate.postingsFormat());
    }
}

This is roughly the high level idea and the key APIs.

Backward Compatibility

One complications from the solution from above is evolution of the formats and backward compatibility. Since we have detouched the plugins from actual codec, the versions (or its equivalents) may need to be included along with the fields info (using attributes() for example). This area is still under exploration right now.

Related component

Plugins

Describe alternatives you've considered

Keep things unchanged

Additional context

• [RFC] Proposal for Moving KNN interfaces to Opensearch core #20050
• https://forum.opensearch.org/t/knn-doesnt-coexist-with-seismic-sparse/27709
• https://forum.opensearch.org/t/codec-error-with-knn/16358

[reta]

@msfroh @navneet1v would appreciate your thoughts folks whenever you have time, is it something along the lines or completely off, thank you!

[navneet1v]

Thanks @reta for adding details here. Here is what I am thinking:
Option 2 is superior and I would recommend that on high level. But I have some questions/suggestions

1. PerFieldStoredFieldsFormat: I am not seeing this in Lucene, please correct me if it is present in Lucene and I am missing something here.
2. CompositeDelegating103PerFieldCodec: This is tricky and this name will struck. I would suggest making it something like OpenSearch103Codec.
3. Will this new codec CompositeDelegating103PerFieldCodec be default codec once the changes rollout?
4. Since we are changing the name of the codec, I will be really interested in how the BWC will work.

[reta]

Thanks a lot @navneet1v

1. PerFieldStoredFieldsFormat: I am not seeing this in Lucene, please correct me if it is present in Lucene and I am missing something here.

Correct, this is not something that Lucene provides but we should be developing in the scope of the our per-field extensions APIs. I included it into the high level APIs to outline the idea (I do have rough prototype).

2. CompositeDelegating103PerFieldCodec: This is tricky and this name will struck. I would suggest making it something like OpenSearch103Codec.

Certainly up for discussion, the flow I have in mind - if there are no plugins that contribute own pieces of PerFieldCodecBuilder, we could just skip it altogether and use the default one.

3. Will this new codec CompositeDelegating103PerFieldCodec be default codec once the changes rollout?

Not necessarily, from the point from above. However we could make it a default as well if there are benefits (simpler implementation).

4. Since we are changing the name of the codec, I will be really interested in how the BWC will work.

👍

[msfroh]

Correct, this is not something that Lucene provides but we should be developing in the scope of the our per-field extensions APIs. I included it into the high level APIs to outline the idea (I do have rough prototype).

Oh -- I'm not entirely convinced that it makes sense to do a PerFieldStoredFieldsFormat.

The reason why Lucene doesn't have one is because stored fields are written row-stride, whereas postings, doc values, norms, etc. are written column-stride. Since you write all the stored fields for a doc in one shot, Lucene doesn't have any hooks to write different columns differently.

In practice, we (OpenSearch) usually only have one stored field that matters: _source. Since we save the XContent bytes in one shot, the lightweight lz4 (or potentially zstd) compression tends to do the trick.

I think I'm okay with only allowing one plugin to modify the stored fields format for a given index. Again, though, I would trigger it through the mapping (or settings) for the index. So, you could have multiple stored field plugins loaded, but each index has to pick which one they would use. @reta, @navneet1v, what do you think?

[reta]

Thanks a lot @msfroh !

In practice, we (OpenSearch) usually only have one stored field that matters: _source. Since we save the XContent bytes in one shot, the lightweight lz4 (or potentially zstd) compression tends to do the trick.

That is correct, it is mostly about _source (and derived _source handling)

I think I'm okay with only allowing one plugin to modify the stored fields format for a given index. Again, though, I would trigger it through the mapping (or settings) for the index. So, you could have multiple stored field plugins loaded, but each index has to pick which one they would use. @reta, @navneet1v, what do you think?

That would be a great design simplification to be fair (since we won't need to build the whole new API trying to squeeze per-field semantic into not really per-field format). So far I see only knn and jvector conflicting there but even with PerFieldStoredFieldsFormat abstraction we won't be able to solve that since this is all about same _source. To conclude - I would +1 to scope StoredFieldsFormat out.

[navneet1v]

Hi @msfroh and @reta ,

Oh -- I'm not entirely convinced that it makes sense to do a PerFieldStoredFieldsFormat.
The reason why Lucene doesn't have one is because stored fields are written row-stride, whereas postings, doc values, norms, etc. are written column-stride. Since you write all the stored fields for a doc in one shot, Lucene doesn't have any hooks to write different columns differently.

Thanks @msfroh for sharing the information and details on StoreFieldFormat. @reta I do believe we should keep StoredFieldsFormat out for now and keep our focus on KNNVectorFormat and DocValuesFormat .

So far I see only knn and jvector conflicting there but even with PerFieldStoredFieldsFormat abstraction we won't be able to solve that since this is all about same _source.

I can add some context on this, actually both jVector and KNN should get out of this StoredFieldsFormat overrides since both of them uses the override for DerivedSource Feature (I am 100% sure about KNN on this, but for jVector my understanding is coming from a discussion with @akash-shankaran). Ideally we should make changes in these plugins to start using the DerivedSource feature from core ref: https://opensearch.org/blog/save-up-to-2x-on-storage-with-derived-source/ . k-NN already has a GH issue for this: opensearch-project/k-NN#3058 . If we have that then _source field override is not needed.

@reta one thing I have and would like to see with this proposal can a knn index using vector field start using zstd compression? Since the zstd compression comes in its own codec.

[reta]

Awesome, thanks a lot @navneet1v and @msfroh , I will update the proposal shortly to reflect changes

@reta one thing I have and would like to see with this proposal can a knn index using vector field start using zstd compression? Since the zstd compression comes in its own codec.

Yeah, so this is a bit different issue and should be solved by #20411, the short version of it is that we do have index codec settings (fe zstd), that would be passed to the wrapping codec fe knn), once we get the change in, we should be able to use zstd easily.

[navneet1v]

Awesome, thanks a lot @navneet1v and @msfroh , I will update the proposal shortly to reflect changes

@reta one thing I have and would like to see with this proposal can a knn index using vector field start using zstd compression? Since the zstd compression comes in its own codec.

Yeah, so this is a bit different issue and should be solved by #20411, the short version of it is that we do have index codec settings (fe zstd), that would be passed to the wrapping codec fe knn), once we get the change in, we should be able to use zstd easily.

Interesting. I need to think over it how it will be working. But I trust you on this. :D

[akash-shankaran]

I can add some context on this, actually both jVector and KNN should get out of this StoredFieldsFormat overrides since both of them uses the override for DerivedSource Feature (I am 100% sure about KNN on this, but for jVector my understanding is coming from a discussion with @akash-shankaran).

@navneet1v The initial version of the derived source feature for jvector will be similar to the KNN implementation opensearch-project/opensearch-jvector#241.
I fully agree with you on converging to the core version of derived sources, to avoid the _source field override. This will also help with design simplification I think.

Also, +1 on scoping the problem down to KnnVectorsFormat and binary fields initially.