Skip to content

Commit 16686a1

Browse files
unamedkrclaude
andcommitted
Fix Hamming attention scale factor: sqrt(pi/2)/m → (pi/2)/m
ClawTeam research findings: 1. Sign-sign agreement estimator uses (pi/2)/m, not sqrt(pi/2)/m sqrt(pi/2) is for QJL random-projection-then-sign (different estimator) 2. RHT seed-invariance: changing seed doesn't change sign agreement (random diagonal cancels in sign comparison) 3. Multi-hash with Permute+RHT K=4: cosine 0.854 for dim=64 Current architecture: int_attn disabled (FP32 attention + 1-bit storage). Scale fix is for future Hamming attention restoration. 33/33 tests pass. PPL unchanged (FP32 attention path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 90848d4 commit 16686a1

2 files changed

Lines changed: 7 additions & 4 deletions

File tree

src/core/tq_turbo_kv.c

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -768,8 +768,11 @@ void tq_turbo_kv_1b_attention_ref(const float* query, const void* kv_cache,
768768
int sketch_dim = dim;
769769
if (sketch_dim < TQ_BK) sketch_dim = TQ_BK;
770770

771-
/* Scale factor uses sketch_dim (total sign bits), not dim */
772-
float scale_factor = sqrtf(TQ_PI_2) / (float)sketch_dim;
771+
/* Scale factor for sign-sign agreement estimator: (pi/2) / m.
772+
* Note: sqrt(pi/2)/m is for random-projection-then-sign (QJL).
773+
* sign-sign (Hamming) uses pi/2 per the arcsin law.
774+
* Currently int_attn is disabled, but fix for future use. */
775+
float scale_factor = TQ_PI_2 / (float)sketch_dim;
773776

774777
/* Step 1: RHT(query) with expansion matching quantize */
775778
float q_rot[TQ_BK];

tests/test_neon_scalar.cpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -295,8 +295,8 @@ TEST(NeonScalarConsistency, HammingAttentionReference) {
295295
tq_turbo_kv_1b_attention_ref(query.data(), kv_blocks.data(),
296296
actual_scores.data(), seq_len, dim);
297297

298-
/* Manual reference computation */
299-
float scale_factor = sqrtf((float)M_PI / 2.0f) / (float)dim;
298+
/* Manual reference computation — sign-sign estimator uses (pi/2)/m */
299+
float scale_factor = ((float)M_PI / 2.0f) / (float)dim;
300300

301301
/* RHT(query) */
302302
std::vector<float> q_rot(dim);

0 commit comments

Comments
 (0)