TextGrad with open-source llm/vlm

[nysp78]

Hi all,

Does anyone know if it's possible to leverage textgrad with open-source llms or vlms such as qwen, llava etc?

thanks!

[vinid]

Hi!

Yes it is possible!

Are you running LLMs locally or through API services?

We support openai compatible endpoints by default so that should be easy to do. If you can give me a more detailed use case I'm happy to help!

[nysp78]

Actually, I'm running LLMs locally using the Transformers library with pipeline or AutoModelForCausalLM. Is it possible to integrate TextGrad in this case?

Thanks for the quick reply!

[vinid]

Yup!

Just define your own engine class: https://github.com/zou-group/textgrad/blob/main/textgrad/engine_experimental/litellm.py

Basically implement something like that but instead of litellm generate you will need to have your forward passes.

It's not going to look super clean, because you might need to put the model inside the engine and it might be a bit leaky as an abstraction. It should work tho

[nysp78]

Thank you very much! Is it possible also with VLMs? Do I need to customize the MultimodalCall?

[vinid]

You just need to customize the two generate calls.

In particular _generate_from_multiple_input. We basically format stuff like openai wants there but you might need to get the image bytes and tokenize them in the method (I think, not sure about your usecase)

[nysp78]

Actually I want to use a model like this from huggingface: https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf

[vinid]

It mostly depends on what you want to do. You can load that model and expose an openai compatible api (with something like vllm).

Or you can write your own engine by implementing those two methods

[nysp78]

Ok, that's cool! I suppose I can define the same custom egine as backward_engine.

Additionally, I would like to use the ImageQAloss, which requires the MultiModalLLM call. I guess it's needed to customize this loss as it requires tg.Variables

Thank you

[vinid]

Yea! Definitely

…

On Sun, Apr 13, 2025, 02:15 Nikos Spyrou ***@***.***> wrote: Ok, that's cool! I suppose I can define the same custom egine as backward_engine as well. Thank you — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARBSS66OSVDJV2TN7RJ3232ZITKFAVCNFSM6AAAAAB254EPSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA3TGNBVGM> . You are receiving this because you commented.Message ID: ***@***.***> *nysp78* left a comment (zou-group/textgrad#171) <#171 (comment)> Ok, that's cool! I suppose I can define the same custom egine as backward_engine as well. Thank you — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARBSS66OSVDJV2TN7RJ3232ZITKFAVCNFSM6AAAAAB254EPSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA3TGNBVGM> . You are receiving this because you commented.Message ID: ***@***.***>

[tomaz-suller]

Hi everyone. @nysp78 did you end up implementing the engine? I'm about to do the same thing but of course wouldn't want to duplicate the work if you're able / willing to open source.

FYI duplicate of #121, related to #145

[kyrie21z]

@tomaz-suller I reproduced the five tutorials of this project using LiteLLM + QwenAPI, and I wonder if it is helpful.
https://github.com/kyrie21z/TextGrad-Tutorial-With-Qwen-LiteLLM/tree/master