cultural-arts/nanoVLM
Updated
sbrzz
sbrzz
AI & ML interests
AI & ML Industry Expert
Recent Activity
updated a model about 6 hours ago
cultural-arts/nanoVLM published a model about 6 hours ago
cultural-arts/nanoVLM liked a model 25 days ago
facebook/bart-large-mnliOrganizations
updated a
model about 6 hours ago
published a
model about 6 hours ago
liked a
model 25 days ago
reacted to codelion's post with 🔥 9 months ago
Post
5302
I wanted to share a technique that's been working really well for recovering performance after INT4 quantization.
Typically, quantizing the LLM to INT4 (unlike say INT8) for inference can incur some accuracy loss. Instead of accepting the quality loss, we used the FP16 model as a teacher to train a tiny LoRA adapter (rank=16) for the quantized model. The cool part: the model generates its own training data using the Magpie technique so no external datasets needed. This is critical because we want to remain as much as possible in the distribution of the model's natural responses.
Last year Apple's foundational models paper (https://arxiv.org/pdf/2407.21075) had proposed a similar technique and found "By using accuracy-recovery LoRA adapters with only rank 16, Alpaca win rate can be improved by 7-18%, GMS8K accuracy is boosted by 5-10%." (page 47).
We saw similar results on Qwen3-0.6B:
Perplexity: 2.40 → 2.09 (only 5.7% degradation from FP16 baseline)
Memory: Only 0.28GB vs 1.0GB for FP16 (75% reduction)
Speed: 3.0x faster inference than FP16
Quality: Generates correct, optimized code solutions
- Pre-trained adapter: codelion/Qwen3-0.6B-accuracy-recovery-lora
- GitHub repo: https://github.com/codelion/ellora
Happy to answer questions about the implementation or help anyone trying to replicate this. The key insight is that quantization errors are systematic and learnable - a small adapter can bridge the gap without negating the benefits of quantization.
Has anyone else experimented with self-distillation for quantization recovery? Would love to hear about different approaches!
Typically, quantizing the LLM to INT4 (unlike say INT8) for inference can incur some accuracy loss. Instead of accepting the quality loss, we used the FP16 model as a teacher to train a tiny LoRA adapter (rank=16) for the quantized model. The cool part: the model generates its own training data using the Magpie technique so no external datasets needed. This is critical because we want to remain as much as possible in the distribution of the model's natural responses.
Last year Apple's foundational models paper (https://arxiv.org/pdf/2407.21075) had proposed a similar technique and found "By using accuracy-recovery LoRA adapters with only rank 16, Alpaca win rate can be improved by 7-18%, GMS8K accuracy is boosted by 5-10%." (page 47).
We saw similar results on Qwen3-0.6B:
Perplexity: 2.40 → 2.09 (only 5.7% degradation from FP16 baseline)
Memory: Only 0.28GB vs 1.0GB for FP16 (75% reduction)
Speed: 3.0x faster inference than FP16
Quality: Generates correct, optimized code solutions
- Pre-trained adapter: codelion/Qwen3-0.6B-accuracy-recovery-lora
- GitHub repo: https://github.com/codelion/ellora
Happy to answer questions about the implementation or help anyone trying to replicate this. The key insight is that quantization errors are systematic and learnable - a small adapter can bridge the gap without negating the benefits of quantization.
Has anyone else experimented with self-distillation for quantization recovery? Would love to hear about different approaches!
updated a
model 10 months ago
published a
model 10 months ago
updated a
model 10 months ago
published a
model 10 months ago
Improve language tag
#1 opened about 1 year ago
by 
lbourdois
Improve language tag
1
#1 opened about 1 year ago
by lbourdois
Improve language tag
1
#1 opened about 1 year ago
by lbourdois
ONNX Demo code
🔥👍 8
17
#4 opened over 1 year ago
by 
cnmoro
liked a
model about 1 year ago
updated 5
models over 1 year ago
sbrzz/TinyLLaVA-SmolLM2-135M-Instruct-dinov2-small-finetuned
Visual Question Answering • 0.2B • Updated • 7
sbrzz/TinyLLaVA-SmolLM2-135M-Instruct-dinov2-small-pretrained
Visual Question Answering • 0.2B • Updated • 5
sbrzz/TinyLLaVA-OpenELM-270M-Instruct-Dinov2-small
Image-Text-to-Text • 0.3B • Updated • 3 • 1
sbrzz/TinyLLaVA-Qwen2.5-0.5B-dinov2-small
Visual Question Answering • 0.5B • Updated • 10
sbrzz/TinyLLaVA-Qwen2.5-1.5B-dinov2-small
Visual Question Answering • 2B • Updated • 5
liked a
model over 1 year ago
updated a
model over 1 year ago