Qwen3 Model Merging Experiments

This repository explores various techniques for merging Large Language Models (LLMs), focusing on the Qwen3-0.6B family. The goal is to combine the strengths of different domain-expert models (e.g., Medical and Code) into a single, versatile model using various merging strategies.

Note

For a detailed walkthrough and insights into the experiments, check out the blog post: Merged & Fine-Tuned: A Case Study on Qwen3 and Domain Fusion

🚀 Merging Techniques

We implement three distinct merging strategies:

1. Simple Linear Merge: A straightforward weighted average of the model weights.
2. Layer-Wise Smooth Merge (Sigmoid): Uses a sigmoid curve to interpolate weights across layers, favoring general knowledge in early layers and domain-specific knowledge in deeper layers.
3. Greedy Merging with Evaluation: A performance-driven approach where layers are selected from different models (Expert A, Expert B, or Base) based on their scores on a small evaluation set of Q&A pairs.

🛠️ Setup

To run the experiments, ensure you have the necessary dependencies installed:

pip install -r requirements.txt

📈 Experiment Details

The experiments are conducted using the following models:

• Base Model: Qwen/Qwen3-0.6B
• Medical Expert: suayptalha/Qwen3-0.6B-Medical-Expert
• Code Expert: suayptalha/Qwen3-0.6B-Code-Expert

Evaluation

The Greedy Merge technique uses a predefined set of 10 Q&A pairs (5 medical, 5 code) to evaluate which model's layer performs best at each position.

📂 Repository Structure

• qwen3_model_merging_experiments.py: The main script containing implementation for all three merging techniques.
• requirements.txt: Python dependencies.
• .gitignore: Files to exclude from the repository.
• README.md: Project documentation.

🤝 Acknowledgments

Special thanks to the authors of the Qwen3 models and the model-merging community for providing the tools and inspiration for these experiments.