feat(java): expose createIndex progress callbacks

[hfutatzhanghb]

Background

Distributed segment builders call Dataset.createIndex(IndexOptions), but the Java binding currently drops the Rust CreateIndexBuilder progress channel. This follow-up to #8090 exposes the existing callback for index creation so Java and Spark callers can observe real stage progress while a segment is still building.

Summary

• add the source-compatible Dataset.createIndex(IndexOptions, IndexBuildProgress) overload
• bridge the callback through JNI into CreateIndexBuilder.progress(...) for committed and uncommitted builds
• release the native dataset field guard while callbacks run, while preserving same-handle create serialization and committed dataset state
• cover real inverted-index stages, callback failures, ignored completion failures, same-Dataset re-entry, committed state refresh, and null validation

Compatibility

• the existing createIndex(IndexOptions) API and no-progress JNI path remain intact
• no storage format or index format changes

Verification

• cargo fmt --manifest-path java/lance-jni/Cargo.toml --all --check
• ./mvnw -Dskip.build.jni=true spotless:check
• ./mvnw -Dskip.build.jni=true -DskipTests compile
• ./mvnw -Dskip.build.jni=true -DskipTests test (compiles Java test sources; tests skipped)
• javap confirms both createIndex overloads

Local Cargo build, test, check, and clippy were intentionally not run on this machine. Rust compilation and end-to-end Java/JNI tests are delegated to GitHub Actions.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

1 fixed / 1 remains. Same-Dataset callback read re-entry is now safe with a queued writer, but nested index creation can still lose the outer Dataset callback context when a later callback runs on another Tokio worker.

A viable revision should capture inherited active Dataset identities when constructing the nested callback wrapper and merge them into every callback invocation, independent of which JNI callback thread executes it.

[lance-gatekeeper]

This map contains only the Dataset registered on the current callback thread. During nested A → B index creation, a later B callback can run on a different Tokio worker; that worker registers B here but never inherits still-active A. If the B callback reads A after an A writer queues, the read waits behind the writer while the outer A create waits for B, causing a deadlock. This is the remaining cross-worker form of the earlier nesting finding.

Capture the inherited active Dataset identities when the nested wrapper is constructed and install or merge them around every callback, regardless of the JNI callback thread.

Reproducer

On this head, I moved the outer/nested Dataset reads in testCreateIndexPreservesOuterDatasetCallbackContextAcrossDatasets from the nested stageStart callback to stageProgress, then ran:

timeout 30s ./mvnw -Dskip.build.jni=true \
  -Djava.io.tmpdir=/home/agent/tmp/gate8823-java-tmp \
  -Dtest='org.lance.index.ScalarIndexTest#testCreateIndexPreservesOuterDatasetCallbackContextAcrossDatasets' test

Maven hung after starting the test and timeout exited 124; the bounded test should complete.