Reaching out to mentors of project 15: GGUF Reader in OpenVINO for direct GGUF Execution

[mostafafaheem]

Hello @cavusmustafa and @ravi9!
I hope you are doing well,

I'm Mostafa, a Computer Science student in my 4th year, and I'm highly interested in contributing to OpenVINO with the GGUF reader project. This project particularly excites me because I have used both llama.cpp and OpenVINO to convert and run fine-tuned LLMs locally in my academic and hobby projects, so working on enabling seamless integration between both would be quite the pleasure.😄

I have experience with multiple Deep Learning frameworks and I have had an obsession with low level ML, IRs and how these frameworks operate under the hood, which makes the prospect of working on this project even more thrilling!

After doing a deep dive over the past few days, I have a few questions regarding the project:

1. If we're going to use llama.cpp APIs for GGUF to GGML conversion and the OpenVINO backend, I believe it's best not to introduce a new third party dependency to OpenVINO, so how should we approach this? Would vendoring be a viable option?
2. Would adding a CMake flag to toggle GGUF support ON/OFF be a welcome addition?
3. Is there a particular performance gold standard I should aim for when profiling and optimizing the reader?
4. I would love to extend GGML's OpenVINO backend to support more operations. Is it fine if I open issues and submit PRs in @ravi9's fork of llama.cpp?

I would love to discuss these ideas further and contribute the best I can to this project. Looking forward to your response!

@adrianboguszewski Can you help connect me with the mentors for further discussion? Thanks in advance!

[arunthakur009]

Hi @cavusmustafa and @ravi9,

I’ve been going through the README, the current GGUF reader implementation in OpenVINO GenAI, and the ggml-openvino backend to understand how the graph translation currently works. The idea of moving from static reconstruction to GGML graph-based translation is very interesting to me.

I had a few clarification questions as I try to understand the expected direction:

1. During the transition to GGUFReaderV2, would the new reader initially coexist with the current static reconstruction approach, or is the goal to fully replace it once functional parity is achieved?
2. For the first milestone, is it expected to prioritize correctness with FP16/BF16 models before handling more complex mixed-quantization formats (like Q4_K_M and Q6_K combinations)?
3. Regarding testing — should output validation focus mainly on logit-level comparison with llama.cpp backend, or is full text-generation parity also expected in early stages?

I’m continuing to explore the code paths and would appreciate any guidance on how you’d suggest scoping the first implementation steps.

Thanks!

[ravi9]

Hi @mostafafaheem :
Thank you for your interest in this project !

1. Similar to how Torch is a dependency for the PyTorch Frontend, we expect llama.cpp to be a dependency for GGUF-Reader-v2. We need llama.cpp APIs to read .gguf, construct the ggml_cgraph, and then pass it to the ggml-openvino logic. We’re open to other ideas that avoid adding a hard dependency! The goal of this project is to evaluate the feasibility of moving the OpenVINO backend from llama.cpp into OpenVINO-GenAI and eventually to OpenVINO.
   The proposed new GGUFReaderV2 should:
   -- 1. call the llama.cpp APIs to produce the ggml_cgraph,
   -- 2. then pass ggml_cgraph to GgmlOvDecoder to produce an ov::Model
   -- 3. then pass ov::Model into the GenAI read_model() pipeline for compilation and inference.

I think, it is better to do it in phases.
Phase 1: Validate if the produced ov::Model in (2) is compatible with GenAI, if not update the GgmlOvDecoder/ggml-openvino.
Phase 2: Move the GgmlOvDecoder/ggml-openvino into OpenVINO GenAI.
Phase 3: Move the GgmlOvDecoder/ggml-openvino into OpenVINO.

2. Yes, a CMake flag to toggle GGUF support ON/OFF would be great !

3. Currently, the goal is to make it functional with several models on Intel CPU, GPU, NPU.

4. Sure, feel free to submit PRs to dev_backend_openvino

cc: @cavusmustafa

Thanks !

[mostafafaheem]

Thank you for taking the time to respond! Given what you said, I will focus my demo on directly calling llama.cpp APIs for now, but I will nevertheless be open to exploring ways of avoiding the hard dependency without compromising on maintainability and overall code quality.

Much thanks for your guidance, looking forward to contribute😊

[ravi9]

Hi @arunthakur009 !
Thank you for your interest in this project !

1. Yes, the new reader will coexist until full functional and performance parity matches.
2. Yes, validate with FP16/BF16 and then to other quantizations
3. Yes, start with logits (deterministic, greedy), then add text-gen parity checks once the pipeline is stable.

cc: @cavusmustafa

Thanks !

[arunthakur009]

Hi @cavusmustafa and @ravi9,

Thank you for your response. I’ve completed the first draft of my GSoC proposal for the GGUF Reader v2 project and would really appreciate your feedback, especially on the architecture and implementation plan.

I’m reaching out a bit close to the deadline due to academic commitments, but even a few brief suggestions would help me refine the proposal significantly.
Click Here

Thanks again for your time!

[arunthakur009]

Hi @cavusmustafa and @ravi9,

I just wanted to share a quick update that I’ve submitted my final proposal for the “GGUF Reader in OpenVINO” project on the GSoC portal.

Thank you both for your guidance and feedback throughout the discussion — it was extremely helpful in refining the architecture and aligning the implementation plan with the GenAI pipeline.

I’m really excited about the possibility of working on this project over the summer. In the meantime, I’ll continue exploring the repository and contributing wherever possible. Please let me know if there’s anything else I should follow up on regarding the proposal.

Best regards,
Arun