⚡ SubQ: First Sub-Quadratic LLM with 12M Token Context
A startup out of Miami claims to have escaped the quadratic attention bottleneck that has defined transformers since 2017 — with a 12-million-token context window to prove it.
📋 Article Summary
- 🔑 The Quadratic Problem — Transformers waste compute on token relationships that don't matter
- ⚙️ Sparse Attention (SSA) — Content-based selection. Linear scaling, not quadratic.
- 📊 Efficiency — 64.5x less compute at 1M tokens. 52x faster than FlashAttention 2.
- ✅ Quality — RULER 95.6%, needle-in-haystack 100% at 2M, SWE-bench 81.8%
- ⚠️ Caveats — Vendor-reported, no open weights, no independent reproduction yet
The Problem
Every transformer LLM since 2017 is bottlenecked by quadratic attention: doubling input length quadruples compute. This is why 1M-token contexts cost $5-25/query and why RAG exists — not because retrieval is better, but because feeding full documents is too expensive.
The Architecture: SSA
SubQ (Miami startup, $29M seed, ex-Meta AI leads) replaces dense attention with content-based sparse selection. Each token learns to pick a small subset of semantically relevant tokens. Full attention math runs only on those pairs.
- • vs Longformer/BigBird — SSA selects by content, not position
- • vs Mamba/RWKV — Exact attention on selected tokens, no approximation loss
- • vs RAG — Full context retained, only computation is sparse
Efficiency Gains
| Context | Compute vs Dense | Speed vs FA2 |
|---|---|---|
| 128K | 8x less | 8x faster |
| 512K | 31x less | 31x faster |
| 1M | 64.5x less | 52x faster |
| 12M | ~1000x less | — |
Quality Benchmarks
- • RULER 128K: 95.6% (vs 94.8% Claude Opus 4.6)
- • Needle-in-Haystack: 100% at 1M and 2M, 98% at 6M and 12M
- • SWE-Bench Verified: 81.8%
- • GPQA Diamond: 85.4% (below frontier ~93%)
Honest caveat: Most benchmarks are vendor-reported. No public weights, no independent reproduction. The architecture is genuine, but real-world 12M-token performance is unverified by third parties. Available in private beta.