CRUD eval problems

[PerfectCui]

Hi,
I have greatly benefited from reading and reproducing your work. Recently, while evaluating with the CRUD dataset, I encountered some phenomena and would like to seek your advice.

In my experiments, I observed that the response length of the LLM appears to be positively correlated with the average chunk length, which in turn leads to fluctuations in metrics such as BLEU. May I ask whether the Dynamic Merging mechanism proposed in your Meta-Chunk method was designed, at least in part, to address this issue?

In addition, I noticed that the reference texts in the CRUD dataset are generally short, whereas the outputs generated by RAG models tend to be much longer. This seems to result in shorter generations having an advantage in evaluation metrics. Does this imply that CRUD, as an evaluation benchmark, may not fully and objectively reflect the true performance ?

I would be very grateful for your guidance, and I truly appreciate your important contributions in this area.

[Robot2050]

Thank you again for your professional experimental exploration! Regarding the content you mentioned, constraints can be imposed through the control variable method of unifying chunk lengths. During our research on CRUD, we adopted a chunk length of 178, as determined by the lumberchunker method, as a benchmark, and ensured that the chunk lengths of all methods remained consistent with it (Appendix D). The dynamic merging mechanism is primarily employed because our Meta-Chunking method may not always divide text into chunks of exactly 178 in length, necessitating the merging of some text chunks.

The name "Meta-Chunk" also originates from this context. We designate the text chunks initially obtained through our method as "Meta-Chunks," which represent text chunks with preliminary logical completeness, laying the foundation for subsequent processes. As a crucial supplement, the dynamic merging mechanism not only preserves the semantic coherence of the content but also flexibly adapts to the length benchmark, enabling more flexible experimental comparisons.

Regarding the observation that "the response length of the LLM appears to be positively correlated with the average chunk length," we have also made new discoveries in our ongoing research. Apart from the aforementioned factor, this also stems from the varying performance of different models. Specifically, Qwen2-7B tends to provide more detailed answers, whereas Qwen2.5-7B leans towards more concise and clear responses. However, this does not impact our experiments significantly, as the chunk lengths, retrieval methods, and generative models employed across different methods remain consistent, ensuring that the final comparisons are unaffected. We are conducting additional experiments on this research topic and plan to release a new round of updates soon. We look forward to your continued attention.