Commit 16686a1
Fix Hamming attention scale factor: sqrt(pi/2)/m → (pi/2)/m
ClawTeam research findings:
1. Sign-sign agreement estimator uses (pi/2)/m, not sqrt(pi/2)/m
sqrt(pi/2) is for QJL random-projection-then-sign (different estimator)
2. RHT seed-invariance: changing seed doesn't change sign agreement
(random diagonal cancels in sign comparison)
3. Multi-hash with Permute+RHT K=4: cosine 0.854 for dim=64
Current architecture: int_attn disabled (FP32 attention + 1-bit storage).
Scale fix is for future Hamming attention restoration.
33/33 tests pass. PPL unchanged (FP32 attention path).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 90848d4 commit 16686a1
2 files changed
Lines changed: 7 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
768 | 768 | | |
769 | 769 | | |
770 | 770 | | |
771 | | - | |
772 | | - | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
773 | 776 | | |
774 | 777 | | |
775 | 778 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
295 | 295 | | |
296 | 296 | | |
297 | 297 | | |
298 | | - | |
299 | | - | |
| 298 | + | |
| 299 | + | |
300 | 300 | | |
301 | 301 | | |
302 | 302 | | |
| |||
0 commit comments