Commit a86a837
fix(gemma4): revert layer_output_scale to residual-separation formula
R13: "x *= los" destroys residual (los=0.0178 → embedding scaled to 0
after 35 layers). Reverted to the original formula:
x = x_input + los * (x_current - x_input)
which preserves the residual and only scales the layer's contribution.
Added TQ_NO_LOS=1 env var for debugging without layer_output_scale.
Still produces garbage — A/B test confirms the issue is in the forward
pass itself (garbage with AND without layer_output_scale, just different
patterns). Waiting for llama.cpp reference output to confirm if the
GGUF file itself is valid.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 8a7825b commit a86a837
1 file changed
Lines changed: 11 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15668 | 15668 | | |
15669 | 15669 | | |
15670 | 15670 | | |
15671 | | - | |
15672 | | - | |
15673 | | - | |
15674 | | - | |
15675 | | - | |
| 15671 | + | |
| 15672 | + | |
| 15673 | + | |
| 15674 | + | |
| 15675 | + | |
| 15676 | + | |
| 15677 | + | |
| 15678 | + | |
| 15679 | + | |
| 15680 | + | |
15676 | 15681 | | |
15677 | 15682 | | |
15678 | 15683 | | |
| |||
15683 | 15688 | | |
15684 | 15689 | | |
15685 | 15690 | | |
15686 | | - | |
| 15691 | + | |
15687 | 15692 | | |
15688 | 15693 | | |
15689 | 15694 | | |
| |||
0 commit comments