36kevin/SCAU-IGEM-Model

🤖 Qwen3 Finetuned Model Series (QLoRA)

This repository contains multiple variants of Qwen3-based models fine-tuned via QLoRA, including base generative models, Thinking models, and RAG companion models (Embedding + Reranker). All models are developed to support iGEM teams and synthetic biology research groups with functionalities such as experimental protocol assistance, iGEM rule explanations, and competition strategy guidance. They are suitable for dialogue, reasoning, and Retrieval-Augmented Generation (RAG) scenarios.

✅ Overall Evaluation Conclusion: After balancing multiple dimensions (performance, reasoning quality, resource consumption), the 4B-parameter base model demonstrates the best overall performance and is recommended as the default choice.

---

📦 Model Overview

Model Type	Parameters	Description
Base Models	0.6B,1.7B,4B,8B,14B	Standard text generation models for general dialogue and instruction following
Thinking Model	14B	Enables "Chain-of-Thought" capability, suitable for complex reasoning tasks
Embedding Model	0.6B	Used for vector retrieval in RAG (sentence embedding)
Reranker Model	0.6B	Used for re-ranking in RAG (cross-encoder style reranking)

---

All models are fine-tuned from the original Qwen3 base weights.

---

⚙️ Finetuning Configuration (QLoRA)

• Quantization: 4-bit (NF4)
• Training Epochs: 4
• Per-device Batch Size: 2
• Gradient Accumulation Steps: 8 (effective batch size = 16)
• Learning Rate Warmup Steps: 4
• LoRA Configuration:
  • rank (r): 8
  • alpha: 256
  • target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
• training framework：transformers + peft + bitsandbytes