Hello again!
Would it be possible to modify the GMP fine tune script to train a LoRA with PEFT for the CLIP VIT-G model? Then merge the LoRA with the model to get a new CLIP-G model?
Chat-GPT seems to think you can do it. But you have some clip module that only has certain clip models from openai, and not the clip G which I think is from LAION?
from peft import get_peft_model, LoraConfig, TaskType # Import LoRA
...
model, preprocess = clip.load(clipmodel, device=device)
# LoRA Configuration
lora_config = LoraConfig(
r=4, # Low-rank dimension
lora_alpha=32,
task_type=TaskType.VISION_LANGUAGE, # Use the appropriate task type
target_modules=["visual.transformer", "transformer"] # Specify target modules for LoRA
)
# Apply LoRA
model = get_peft_model(model, lora_config)
Hello again!
Would it be possible to modify the GMP fine tune script to train a LoRA with PEFT for the CLIP VIT-G model? Then merge the LoRA with the model to get a new CLIP-G model?
Chat-GPT seems to think you can do it. But you have some clip module that only has certain clip models from openai, and not the clip G which I think is from LAION?