How can I use a dynamic tokenizer?

[tezlm]

I have a document with a lang field and a content field. How would I tokenize content based on the language?

It doesn't look like tokenizers can be swapped out as needed:

Tokenizers are currently configured and initialized per field, there's no dynamic tokenizer mechanism.

You could have a tokenizer with a language detection or have seperate fields.

#1708 (comment)

[tezlm]

currently doing a hacky workaround where i index lang:actual content here then do split_at in the tokenizer