Question about class-agnostic setting / comparison to pure 3D baseline

[Xin200203]

Hi, thank you very much for releasing the code and for this excellent work.

I have been carefully reading the paper and studying the implementation. I am currently experimenting with a related architecture, and I found the design choices in SegDINO3D very inspiring.

I would like to ask a question about the class-agnostic setting.

In my own experiments with a similar 2D-3D fusion design, I observed that injecting 2D features can sometimes slightly improve semantic/category-related cues, but may also hurt mask boundary quality (for example, lower purity/coverage at the instance-mask level), especially when compared with a pure 3D baseline. This made me wonder whether SegDINO3D has also been evaluated in a class-agnostic formulation, where the focus is more on instance mask quality itself rather than category prediction.

So my question is:

Did you ever try a class-agnostic version of SegDINO3D?

If yes, how did it compare with a pure 3D baseline?

More specifically, did 2D injection still improve the results in that setting, or did you observe any trade-off on mask boundary quality / instance completeness?

I am asking mainly to better understand whether the 2D features in your framework are also beneficial for class-agnostic instance segmentation, or whether their main benefit is concentrated on the class-aware setting.

I would greatly appreciate any comments, even informal observations, if this was not included in the paper.

Thank you again for your great work.

[JinyuanQu]

Thank you for your interest in our work.

We have not conducted dedicated training for the class-agnostic setting. However, you may try directly using the checkpoint we provide to evaluate under this setting.

From my perspective, I would still expect performance improvements. Even when ignoring the impact of category estimation errors, evaluation under the class-agnostic setting still requires computing a score for each query. The introduction of 2D features can help determine whether a region corresponds to an instance. For example, when the 3D point cloud features alone are ambiguous, the 2D prior can provide a more reliable signal. Regarding the mask boundary issue you mentioned, in the 3D point cloud segmentation benchmarks we use, most methods perform segmentation directly on superpoints. The ground truth annotations are also defined on this basis. This design significantly simplifies the boundary problem. In practice, I rarely observe baseline methods making boundary segmentation mistakes. More commonly, they fail to recognize that an object exists or produce an incorrect category prediction. After introducing 2D features, we did not observe degradation in mask boundary quality.

I would suggest checking the following aspects. First, whether the 2D model you use can provide strong and discriminative features. Second, whether the mapping during the feature injection process is accurate. For instance, if there is a projection error when extracting 2D features for points located on the boundary of a 3D object, background 2D features may be injected instead, which could harm performance. In fact, this process places certain requirements on the accuracy of the scene geometry.