Relational DB (cognee_db) grows unbounded — no eviction on results/queries tables

[king-sheol]

Description

The internal SQLite database (cognee_db) grows without bound when Cognee is used as a long-running service with automated search queries. There is no eviction, TTL, or size cap on the results and queries tables, and SQLite does not auto-VACUUM, so the file only grows — never shrinks.

This is a companion to #2538 (unbounded log growth). Same root pattern — write-only accumulation with no rotation — but in the relational store rather than log files.

Impact

On our production system running Cognee as an HTTP daemon with automated monitoring and ingestion pipelines:

Metric	Value
DB file size	28 GB
Period	9 days
Growth rate	~3 GB/day
results rows	42,152
queries rows	42,174
nodes rows	23,450
edges rows	48,029
data rows	1,409
SQLite freelist pages	0 (no reclaimable space)

The 28 GB file filled the disk to 89%, causing memory pressure (the daemon had to swap), degraded search latency, and eventually a cascading service restart.

Root cause

modules/search/operations/log_result.py and log_query.py append a row on every search call with no cleanup:

# log_result.py — write-only, no eviction
async def log_result(query_id, result, user_id):
    db_engine = get_relational_engine()
    async with db_engine.get_async_session() as session:
        session.add(Result(value=result, query_id=query_id, user_id=user_id))
        await session.commit()

Result.value is a Text column that stores serialized search results including full node payloads. At ~4,700 queries/day (typical for a daemon with health probes and automated ingestion), the table reaches tens of gigabytes within days.

Additionally, the nodes and edges tables mirror graph data that already lives in the graph DB (FalkorDB/Neo4j), creating a second unbounded copy.

Environment

• Cognee v0.5.5 (same issue likely exists in v0.5.7 — the log rotation PR fix: log rotation, safe default dir, and opt-out file logging #2541 did not touch the relational layer)
• VECTOR_DB_PROVIDER=falkor (graph + vectors in FalkorDB)
• HTTP daemon mode (~4,700 search queries/day from automated pipelines)
• SQLite as the default relational backend

Proposed solution

Short-term (cache eviction):

• Add a configurable max row count or TTL for results and queries tables (e.g., COGNEE_MAX_CACHED_RESULTS=10000, COGNEE_QUERY_HISTORY_TTL_DAYS=30)
• FIFO eviction: delete oldest rows when the cap is exceeded
• Run eviction as part of the search call or as a periodic background task

Medium-term (VACUUM):

• Periodic auto-VACUUM or PRAGMA auto_vacuum = INCREMENTAL to reclaim space from deleted rows
• Alternatively, COGNEE_DB_VACUUM_INTERVAL env var for daemon deployments

Optional:

• Make query/result logging opt-out via COGNEE_LOG_QUERIES=false for daemon deployments that don't need search history
• Consider whether nodes/edges tables need to exist when an external graph DB is configured — they duplicate data and contribute to growth

Workaround

Manual cleanup (requires stopping the daemon):

# Export tracking data
sqlite3 cognee_db ".dump data" > /tmp/tracking_backup.sql
sqlite3 cognee_db ".dump datasets" >> /tmp/tracking_backup.sql

# Delete and restart — daemon recreates empty DB
rm cognee_db
systemctl restart cognee-daemon

# Restore tracking
grep "^INSERT" /tmp/tracking_backup.sql | \
  sed 's/INSERT INTO/INSERT OR REPLACE INTO/' | \
  sqlite3 cognee_db

Or periodic VACUUM via cron (does not require restart, but only reclaims space from deleted rows):

0 3 * * 0 sqlite3 /path/to/cognee_db "DELETE FROM results WHERE created_at < datetime('now', '-7 days'); DELETE FROM queries WHERE created_at < datetime('now', '-7 days'); VACUUM;"

[Vasilije1990]

@king-sheol logs I get and it is definitely not ideal if the server instance is hosted locally and causes increase of such size.

On the other hand, sqlite is not there for production use-cases and I would encourage you to deploy a RDS or a postgres instance.

I will anyhow look into this, but I definitely do not want to by default make people lose metadata just because.

[king-sheol]

@Vasilije1990 Thanks for the quick response!

Totally agree that Postgres is the right call for production — and we'll likely migrate to it at some point.

That said, I think the core issue is independent of the DB engine: log_result() and log_query() append on every search call with no eviction. On Postgres, the same workload would produce the same 42K rows of cached results — the table growth is unbounded regardless of backend. Postgres handles it more gracefully (autovacuum, TOAST compression), but the data accumulation pattern is the same.

I'm definitely not suggesting deleting metadata by default. The data and datasets tables are valuable and should be preserved. The concern is specifically about:

• results — a search result cache (42K rows in 9 days). This is ephemeral by nature and safe to evict with a configurable TTL or max row count.
• queries — search history log (42K rows). Same — useful for analytics, but should have a retention policy.

A simple opt-in like COGNEE_RESULT_CACHE_TTL_DAYS=30 or COGNEE_MAX_CACHED_RESULTS=10000 would solve it without any risk of losing important data.

For now, we're running a weekly cron to clean up old results/queries + VACUUM, which works fine as a workaround.

[Vasilije1990]

@king-sheol yup, understood. My proposal is to auto-disable them by default. In that case we avoid ugly logic in the SDK itself that counts and checks how many rows we have and cleans up. I will investigate a bit more and talk to the team to find an answer on this one

[king-sheol]

@Vasilije1990 Auto-disabling by default sounds like the cleanest approach — no eviction logic, no row counting, just a simple flag.

Happy to submit a PR for this if it helps. The change is small:

• Add COGNEE_LOG_QUERIES env var (default: false)
• Guard log_result() and log_query() with an early return when disabled
• Document the flag in .env.template

Let me know if you'd like us to take a crack at it, or if you'd prefer to handle it internally.

[Vasilije1990]

@king-sheol already handled it and added an index: #2549

Let me get the team to review and will keep you posted

Would also require an alembic migration