MikeyBeez/HRS: HRS v1.0: 15-Configuration Ablation Study

Hierarchical Routed Sinkformer Geometry-Shaped Representations for Compute-Adaptive Language Modeling Results 15 configurations across 5 experimental axes on WikiText-103: | Config | Best Val PPL | vs Baseline | |--------|-------------|-------------| | dense_baseline | 23.83 | -- | | full_hrs (v1) | 10.64 | -55.4% | | full_hrs_refined (v1) | 9.19 | -61.4% | | full_hrs_trc (v1) | 9.43 | -60.4% | | v2_attn_conv_peer | 24.1 | +1.1% | | v3_full | 9.46 | -60.3% | | v4_full | 8.3 | -65.2% | | v5_replace | 10.4 | -56.3% | Key Findings Routing alone hurts; routing + engram compression is transformative Fixed structural priors (v2) cannot substitute for learned routing Expert capacity is not the bottleneck (v3: 262K experts = 4 experts) Expert role matters (v4: unconditional PEER + 3-tier routing = best result) Prepending engrams outperforms in-place replacement (v5) by 25% Phase 5 (refinement) consistently regresses -- 4-phase protocol is optimal Hardware RTX 5070 Ti (16GB VRAM), 32GB RAM WikiText-103, seq_len=512 ~6-8 hours per experiment Paper See HRS_paper.pdf and HRS_paper.md in the repo.