Skip to content

perf(for-you): bound my_artist_affinity and follow_set by recency#806

Merged
dylanjeffers merged 1 commit into
mainfrom
perf/bound-affinity-follow-set
May 13, 2026
Merged

perf(for-you): bound my_artist_affinity and follow_set by recency#806
dylanjeffers merged 1 commit into
mainfrom
perf/bound-affinity-follow-set

Conversation

@dylanjeffers
Copy link
Copy Markdown
Contributor

Summary

Builds on #805 (which capped `my_saved_artists`). After that fix the endpoint still times out on prod for power users — the remaining unbounded CTEs are scanning the user's full history every request:

  1. `my_artist_affinity` unions saves + reposts + plays for the caller. `plays` is the biggest table by far — a heavy listener can have hundreds of thousands of rows, all scanned on every request. Cap each source to most recent N: 200 saves, 200 reposts, 500 plays.

  2. `follow_set` is every user the caller follows; for a power-user with thousands of follows this becomes a wide hash join against every recent-track upload. Cap to 500 most-recently followed.

Recency is the right axis on all three: old engagement is a weak signal of current taste, and the bounds match the magnitude of the hidden cost (plays >> saves ≈ reposts).

Diff (CTEs)

```sql
follow_set AS (
SELECT followee_user_id AS user_id FROM follows
WHERE follower_user_id = @userid AND is_current AND NOT is_delete
ORDER BY created_at DESC LIMIT 500 -- new
),
my_artist_affinity AS (
SELECT t.owner_id, LN(1 + COUNT(*)) AS affinity
FROM (
(SELECT save_item_id ... ORDER BY created_at DESC LIMIT 200) -- new
UNION ALL
(SELECT repost_item_id ... ORDER BY created_at DESC LIMIT 200) -- new
UNION ALL
(SELECT play_item_id ... ORDER BY created_at DESC LIMIT 500) -- new
) eng JOIN tracks t ON ... GROUP BY t.owner_id
),
```

Test plan

  • ✅ All 9 `TestV1FeedForYou_*` tests pass locally. Fixtures have <200 saves/<200 reposts/<500 plays/<500 follows so the caps don't kick in and observable behavior is unchanged.
  • ✅ `go build ./api/...` / `go vet ./api/...` clean.
  • After deploy: `/v1/users/eYZmn/feed/for-you?user_id=eYZmn&limit=5` (notjulian, deep history) — currently times out at the Cloudflare upstream (>120s). Target: <2s.

Follow-ups

Parallel EXPLAIN ANALYZE work happening to verify the bound shifts the cost as expected and to flag any missing indexes.

🤖 Generated with Claude Code

After capping my_saved_artists (PR #805), the endpoint still times out
on prod for power users. Two remaining unbounded CTEs are doing full
scans of per-user history:

* my_artist_affinity unions saves+reposts+plays for the user. A heavy
  listener can have hundreds of thousands of play rows; the inner
  scan dwarfs the rest of the query. Cap each source to its most
  recent N: 200 saves, 200 reposts, 500 plays.
* follow_set is the followee set for the user; a power user can
  follow several thousand artists, which then has to join against
  every recent-track upload in cand_in_network. Cap to 500 most
  recently followed.

Recency is the right axis on all three: older history is a weaker
signal of current taste, and the bounds match the magnitude of the
hidden costs (plays >> saves ≈ reposts). Existing For You tests have
fixtures well under all caps and still pass unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@dylanjeffers dylanjeffers merged commit 276fdeb into main May 13, 2026
4 checks passed
@dylanjeffers dylanjeffers deleted the perf/bound-affinity-follow-set branch May 13, 2026 04:11
dylanjeffers added a commit that referenced this pull request May 13, 2026
## Summary

Retiring the dedicated For You feed endpoint. The clients are being
switched to use \`/v1/users/{id}/recommended-tracks\` instead — the same
endpoint that already powers the Explore page's For You section and
works fine in production. See companion PR: AudiusProject/apps#14301.

## Why

The custom \`/feed/for-you\` endpoint had repeated issues since it
shipped:

* **Auth gate bug** (fixed in #804) — global authMiddleware rejected
unsigned \`user_id\` requests, making the endpoint unreachable from the
web RC.
* **Perf** — even after #805 and #806 capped the \`my_saved_artists\`,
\`my_artist_affinity\`, and \`follow_set\` CTEs, EXPLAIN on prod showed
the \`similar_artists\` self-join still produced a 301M-row merge for
power users (and a fixed ~12s \`track_trending_scores\` scan for *every*
user due to a missing partial index). The endpoint never reliably
completed within Cloudflare's 100s upstream limit for power users.
* **Duplication** — the response shape (ranked track list for the
signed-in user) is already what \`/recommended-tracks\` returns. Two
endpoints solving the same problem isn't worth maintaining.

Consolidating on the working endpoint is simpler than continuing to
optimize the custom one.

## Removed

| File | What |
|---|---|
| \`api/v1_users_feed_for_you.go\` | Handler + the 200-row
candidate-pool SQL (4 candidate sources, similar_artists CF, diversity
pass) |
| \`api/v1_users_feed_for_you_test.go\` | 9 unit tests |
| \`api/server.go\` (1 line) | Route registration |
| \`api/auth_middleware.go\` (~10 lines) | The \`/feed/for-you\`
exemption from #804 — no longer needed |
| \`api/swagger/swagger-v1.yaml\` (~70 lines) | The endpoint's swagger
entry |

## Test plan

- ✅ \`go build ./api/...\` clean
- ✅ \`go vet ./api/...\` clean
- ✅ All remaining \`TestV1UsersFeed*\` / \`TestAuth*\` tests pass
locally
- After merge + deploy + AudiusProject/apps#14301 deploy: Feed → For You
tab on the web RC should show the same recommended tracks as Explore's
For You section.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant