Fine-tuneJune 29, 2026 · 4 min read
🔬 Qwythos 9B: Claude Mythos Distillation into Open Weights
A 9B model fine-tuned on 500M+ tokens of Claude Mythos reasoning traces gains +34 points on MMLU and native tool calling — showing how far targeted distillation can push small open models.
📋 Article Summary
- 🔑 What Is Qwythos — Full fine-tune of Qwen3.5-9B on Claude Mythos traces, Apache 2.0
- 📈 Performance Gains — +34 MMLU, +30 gsm8k-strict, +19 gsm8k-flex
- 🔧 Capabilities — 1M context via YaRN, native function calling, optional vision
- ⚠️ Trade-offs — GPQA regression, narrow improvement domain, not general-purpose
- 💡 Why It Matters — Targeted distillation as a force multiplier for small models
What It Is
Qwythos-9B (Empero AI) is a full-parameter fine-tune of Qwen3.5-9B trained on 500M+ chain-of-thought reasoning traces. Ships with 1M context (YaRN rope scaling from 262K native), native function calling, and optional vision projector. Apache 2.0.
Performance vs Base
| Metric | Qwen3.5-9B | Qwythos-9B | Delta |
|---|---|---|---|
| MMLU | 23.2% | 57.5% | +34.3 |
| gsm8k strict | 51.0% | 81.0% | +30.0 |
| gsm8k flex | 67.0% | 86.0% | +19.0 |
| GPQA Diamond | 63.0% | 58.0% | -5.0 |
Key Observations
- • The massive MMLU gain (+34) is partly because base Qwen3.5-9B is unusually low on MMLU
- • GPQA Diamond decreases — fine-tuning trades some graduate science for test performance
- • Independent testing: 100-150 tok/s on RTX 4090, fits 6-8GB VRAM at Q4_K_M
- • Tool-calling: 7/7 correctness on Python + web search harness
Key takeaway: Targeted fine-tuning on high-quality traces dramatically improves small model performance on structured tasks. Best for math, code, and tool-use workflows at the 9B scale.