feat: add Czech stemmer and stopwords

[jonasjancarik]

What

Adds Czech language support to Tantivy stemming and stopwords.

• enable tantivy-stemmers Czech algorithm feature (czech_dolamic_aggressive)
• add Language::Czech to stemmer language enum
• map Czech stemmer to tantivy_stemmers::algorithms::czech_dolamic_aggressive
• add Czech stopwords list (source: stopwords-iso/stopwords-cs, MIT)
• wire StopWordFilter::new(Language::Czech) to Czech stopwords
• add regression tests for Czech stemmer and Czech stopwords

Why

ParadeDB needs Czech support in both stemmer and stopwords_language, same model as existing Polish integration.

Tests

• cargo test tokenizer::tests::test_cs_stemmer --lib
• cargo test tokenizer::stop_word_filter::tests::test_stop_word_czech_language --lib
• cargo test tokenizer::tests::test_pl_stemmer --lib

[CLAassistant]

All committers have signed the CLA.

[philippemnoel]

Thanks for this! Could you please sign the CLA and fix the CI? We'll be eager to review. It might also be worth raising this upstream in Tantivy

Accidentally approved, sorry!

[jonasjancarik]

On the rustfmt red: I checked the failing merge-ref diff and it appears unrelated to the Czech change itself. The failure is a formatting diff in src/lib.rs on the merged PR ref, not in the stemming/stopwords wiring added here.

I'm intentionally leaving that out of this feature PR to keep the scope narrow. If you prefer, I can follow up with a separate formatting-only fix or rebase once the base branch is aligned.

[jonasjancarik]

Fixed the rustfmt failure in src/lib.rs in 1bd3462.

This diff looks unrelated to the Czech stemmer/stopwords change itself, but I included it here to get the branch back to a green formatter state.

Local checks run:

• cargo +nightly fmt --all -- --check
• cargo test tokenizer::tests::test_cs_stemmer --lib

[jonasjancarik]

Tracked the current CI failure.

I pushed 416a57a to fix the branch-specific break: the new Czech stopword test was importing Language unconditionally, but Language is only re-exported when the stemmer feature is enabled. That is what broke test-quickwit / test-none under --no-default-features.

Local verification:

• cargo test --no-run --workspace --features mmap,quickwit,failpoints --no-default-features ✅

There is still a separate --no-default-features failure in example compilation (tempfile / Index::create_in_dir), but that reproduces on origin/main too, so it looks pre-existing and unrelated to this PR.

I left that baseline issue untouched in this feature branch.

[philippemnoel]

Thanks. We're pretty swamped with other issues, but we'll take a look once we get a chance! I'd recommend putting this in upstream as well to see if you can get feedback there, and if it's interesting to the broader Tantivy project

[philippemnoel]

@stuhood Feel free to merge and/or review as well if you want, and then we can have the contributor update the hash and merge this paradedb/paradedb PR

[jonasjancarik]

Thanks. We're pretty swamped with other issues, but we'll take a look once we get a chance! I'd recommend putting this in upstream as well to see if you can get feedback there, and if it's interesting to the broader Tantivy project

Thanks, I can try upstream, but I expect this exact patch shape to get the same reaction as the upstream Polish proposal in quickwit-oss/tantivy#2746. The maintainer feedback there was mainly about not wanting to keep both rust-stemmers and tantivy-stemmers; they did say a full switch to tantivy-stemmers might be acceptable in principle: comment. That may be doable, but I'm not sure how quickly upstream would want to take on that larger change.

There is also an open Czech addition on the rust-stemmers side in CurrySoftware/rust-stemmers#22, which also doesn't seem to have gone anywhere.

For this PR I mainly wanted to follow the pattern ParadeDB had already accepted for Polish in paradedb/tantivy#82 and then integrated on the product side in paradedb/paradedb#3645. So this is meant as the same fork-local approach already used for Polish, just extended to Czech.

[philippemnoel]

Thanks. We're pretty swamped with other issues, but we'll take a look once we get a chance! I'd recommend putting this in upstream as well to see if you can get feedback there, and if it's interesting to the broader Tantivy project

Thanks, I can try upstream, but I expect this exact patch shape to get the same reaction as the upstream Polish proposal in quickwit-oss/tantivy#2746. The maintainer feedback there was mainly about not wanting to keep both rust-stemmers and tantivy-stemmers; they did say a full switch to tantivy-stemmers might be acceptable in principle: comment. That may be doable, but I'm not sure how quickly upstream would want to take on that larger change.

There is also an open Czech addition on the rust-stemmers side in CurrySoftware/rust-stemmers#22, which also doesn't seem to have gone anywhere.

For this PR I mainly wanted to follow the pattern ParadeDB had already accepted for Polish in paradedb/tantivy#82 and then integrated on the product side in paradedb/paradedb#3645. So this is meant as the same fork-local approach already used for Polish, just extended to Czech.

Makes sense. We should do that clean up and move to tantivy-stemmers altogether, and contribute it upstream

[stuhood]

As soon as this is resolved: #111 (comment) ... we can merge this. Thanks!

[jonasjancarik]

Ok, link with the exact blob SHA for the stopwords list added in fc4c629

[stuhood]

Thanks!