feat: add opt-in event retention purge (#359) by Justxd22 · Pull Request #412 · cameri/nostream

Justxd22 · 2026-04-09T11:55:00Z

Added limits.event.retentionDays to settings.yaml.
purge expired, soft-deleted, and older-than-retention events from maintenance worker loop
Added efficient batch-delete query in EventRepository that handles retention, NIP-40 (expiration), and NIP-09 (soft-deletes) in a single operation.
keep behavior opt-in: no purge when retentionDays is unset/non-positive

Description

The logic is hooked into the existing heartbeat of the MaintenanceWorker:

Every 60 seconds, the worker wakes up to check for pending invoices.
It now also checks the retentionDays setting.
If enabled, it executes a single SQL DELETE query targeting:
- Events where created_at is older than the retention threshold.
- Events with an expired expires_at timestamp (NIP-40).
- Events marked as deleted via deleted_at (NIP-09).
Results are logged to the console using [Maintenance] tags for easy monitoring.

Add the following to your .nostr/settings.yaml:

limits:
    event:
        retentionDays: 30 # Set to 0 or comment out to disable.

Related Issue

#359

@Justxd22 Would be good to have a setting so this is off if a date is not set. Making it default to 30 day would be a breaking change.

How Has This Been Tested?

i made test script that add events using wscat with timestamp older than 30 days => wait for next 60 sec and check logs
i disabled the 30 days config and used same script => no delete
re-added 30 days config and restarted => on startup it checked older events and auto deleted them

Screenshots (if appropriate):

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my code changes.
All new and existing tests passed.

cameri · 2026-04-09T12:42:02Z

Please recheck the deletion conditions. There's a query that potentially wipes the events table. That would be catastrophic.

cameri

I think some changes are required to move forward, but mostly looking pretty good!

resources/default-settings.yaml

CONFIGURATION.md

src/app/maintenance-worker.ts

src/repositories/event-repository.ts

Justxd22 · 2026-04-09T17:05:11Z

refactored to new service MaintenanceService in src/services/maintenance-service.ts
the service is called from the existing MaintenanceWorker heartbeat.
purging defaults to -1 now, Also any non-positive value (0 or -1) disables the purge logic
rewrote the sql query to be logically grouped

    return this.masterDbClient('events')
      .where(function () {
        this.where(function () {
          this.whereNotNull('expires_at').andWhere('expires_at', '<', now)
        })
        .orWhereNotNull('deleted_at')
        .orWhere('event_created_at', '<', retentionLimit)
      })
      .del()

i've missed the deletion conditions BUG during my tests as i've been adding only 1 event and testing with it, Good catch @cameri !

src/repositories/event-repository.ts

resources/default-settings.yaml

src/repositories/event-repository.ts

cameri · 2026-04-10T02:51:41Z

Looking good! Left just a few comments.

Copilot

Pull request overview

Adds an opt-in event retention/purge mechanism that runs from the existing MaintenanceWorker heartbeat, enabling operators to automatically delete expired (NIP-40), soft-deleted (NIP-09), and older-than-retention events via a repository-level delete operation.

Changes:

Introduces limits.event.retentionDays setting (default disabled) and documents it.
Adds MaintenanceService.clearOldEvents() and wires it into MaintenanceWorker schedule loop.
Implements EventRepository.deleteExpiredAndRetained() and adds unit tests for worker/service/repository behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/unit/services/maintenance-service.spec.ts	Adds unit tests for retentionDays gating and error handling in `MaintenanceService`.
test/unit/repositories/event-repository.spec.ts	Adds query/behavior tests for `deleteExpiredAndRetained`.
test/unit/app/maintenance-worker.spec.ts	Adds unit tests ensuring the worker calls maintenance even when payments are disabled.
src/services/maintenance-service.ts	New service that reads settings and triggers event purging.
src/repositories/event-repository.ts	Adds a single delete query handling expiration, soft-deletes, and retention cutoff.
src/factories/maintenance-worker-factory.ts	Wires `MaintenanceService` into `MaintenanceWorker`.
src/factories/maintenance-service-factory.ts	New factory for constructing `MaintenanceService` with `EventRepository`.
src/app/maintenance-worker.ts	Invokes `maintenanceService.clearOldEvents()` on each schedule tick.
src/@types/settings.ts	Adds `retentionDays?: number` to event limits settings.
src/@types/services.ts	Adds `IMaintenanceService` interface.
src/@types/repositories.ts	Adds `deleteExpiredAndRetained()` to `IEventRepository`.
resources/default-settings.yaml	Adds default `limits.event.retentionDays: -1` (disabled).
CONFIGURATION.md	Documents `limits.event.retentionDays` behavior and default.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/app/maintenance-worker.ts

src/repositories/event-repository.ts

Copilot

Pull request overview

Adds an opt-in event retention/expiry purge to reduce long-term storage and index load by periodically deleting expired (NIP-40), soft-deleted (NIP-09), and retention-aged events as part of the existing maintenance heartbeat.

Changes:

Introduces limits.event.retentionDays configuration (default disabled) and documents it.
Adds a maintenance service and wires it into MaintenanceWorker’s scheduled loop.
Implements a single-query purge method in EventRepository and adds unit tests around the new behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/unit/services/maintenance-service.spec.ts	Adds unit coverage for retentionDays gating and error handling in the maintenance service.
test/unit/repositories/event-repository.spec.ts	Adds unit coverage asserting generated SQL and retentionDays guard behavior for the purge query.
test/unit/app/maintenance-worker.spec.ts	Adds unit coverage ensuring maintenance runs regardless of payments being enabled.
src/services/maintenance-service.ts	Implements `MaintenanceService.clearOldEvents()` with retentionDays gating and logging.
src/repositories/event-repository.ts	Adds `deleteExpiredAndRetained()` delete query combining retention, expiry, and soft-deletes.
src/factories/maintenance-worker-factory.ts	Wires the new maintenance service into the worker factory.
src/factories/maintenance-service-factory.ts	Adds factory to construct `MaintenanceService` with an `EventRepository`.
src/app/maintenance-worker.ts	Invokes maintenance purge on the existing schedule before invoice processing.
src/@types/settings.ts	Extends `EventLimits` with `retentionDays?: number`.
src/@types/services.ts	Adds `IMaintenanceService` interface.
src/@types/repositories.ts	Extends `IEventRepository` with `deleteExpiredAndRetained()`.
resources/default-settings.yaml	Adds `limits.event.retentionDays: -1` default (disabled).
CONFIGURATION.md	Documents the new `limits.event.retentionDays` setting.

Comments suppressed due to low confidence (1)

src/app/maintenance-worker.ts:44

clearOldEvents() is awaited before invoice processing. If the purge DELETE takes a long time, it will delay invoice updates and can also increase the chance that overlapping onSchedule() runs occur (since setInterval doesn't wait). Consider adding a re-entrancy guard (skip if a run is in progress) and/or running purge and invoice checks concurrently so invoice processing isn't blocked by a large purge.

  private async onSchedule(): Promise<void> {
    const currentSettings = this.settings()

    await this.maintenanceService.clearOldEvents()

    if (!path(['payments','enabled'], currentSettings)) {
      return
    }

    const invoices = await this.paymentsService.getPendingInvoices()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/services/maintenance-service.ts

src/repositories/event-repository.ts

CONFIGURATION.md

cameri · 2026-04-10T03:05:20Z

Please address Copilot comments

Copilot

Pull request overview

This PR adds an opt-in event retention purge that runs from the existing MaintenanceWorker heartbeat, deleting batches of expired (NIP-40), soft-deleted (NIP-09), and older-than-retention events via a single repository query.

Changes:

Introduces MaintenanceService.clearOldEvents() and wires it into MaintenanceWorker’s scheduled loop.
Adds EventRepository.deleteExpiredAndRetained() to batch-delete up to 1000 qualifying events per run, with optional kind/pubkey whitelists.
Adds settings/types/docs for limits.event.retention.maxDays plus unit tests covering worker/service/repository behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/unit/services/maintenance-service.spec.ts	Adds unit tests for retention-driven purge orchestration + logging/error handling.
test/unit/repositories/event-repository.spec.ts	Adds query-string tests for the new batched purge query and whitelist behavior.
test/unit/app/maintenance-worker.spec.ts	Ensures scheduled loop always calls maintenance, regardless of payments config.
src/services/maintenance-service.ts	New service that reads retention config and triggers repository purge.
src/repositories/event-repository.ts	Implements the batched delete query + purge count mapping.
src/factories/maintenance-worker-factory.ts	Wires `MaintenanceService` into the worker factory.
src/factories/maintenance-service-factory.ts	New factory creating `MaintenanceService` + `EventRepository`.
src/app/maintenance-worker.ts	Adds maintenance execution to the schedule + overlap protection.
src/@types/settings.ts	Adds retention config types under `limits.event.retention`.
src/@types/services.ts	Introduces `IMaintenanceService`.
src/@types/repositories.ts	Adds purge options/counts types + repository method signature.
resources/default-settings.yaml	Adds default retention config (disabled via `maxDays: -1`).
CONFIGURATION.md	Documents new retention settings.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/app/maintenance-worker.ts

src/repositories/event-repository.ts

+      .where(function () {
+        this.where('expires_at', '<', now)
+          .orWhereNotNull('deleted_at')
+          .orWhere('event_created_at', '<', retentionLimit)
+      })
+      .modify((query) => {
+        if (Array.isArray(options?.kindWhitelist) && options.kindWhitelist.length > 0) {
+          query.whereNot((builder) => {
+            options.kindWhitelist.forEach((kindOrRange) => {
+              if (Array.isArray(kindOrRange)) {
+                builder.orWhereBetween('event_kind', kindOrRange)
+              } else {
+                builder.orWhere('event_kind', kindOrRange)
+              }
+            })


src/@types/repositories.ts

resources/default-settings.yaml

CONFIGURATION.md

Copilot

Pull request overview

Adds an opt-in maintenance feature to purge events that are expired (NIP-40), soft-deleted (NIP-09), or older than a configured retention window, wired into the existing MaintenanceWorker heartbeat.

Changes:

Introduces MaintenanceService.clearOldEvents() and wires it into MaintenanceWorker’s scheduled loop.
Adds EventRepository.deleteExpiredAndRetained() with a batch-limited delete query and whitelist exclusions.
Extends settings/types/docs/default settings and adds unit tests for worker/service/repository behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/unit/services/maintenance-service.spec.ts	Adds unit tests for retention-enabled/disabled and error handling paths.
test/unit/repositories/event-repository.spec.ts	Adds SQL-string tests for the purge delete query and whitelist behavior.
test/unit/app/maintenance-worker.spec.ts	Adds unit tests ensuring scheduled runs invoke maintenance and respect payments enabled/disabled.
src/services/maintenance-service.ts	New service implementing settings-driven purge orchestration and logging.
src/repositories/event-repository.ts	Implements batch purge query + returned counts for deleted/expired/retained rows.
src/factories/maintenance-worker-factory.ts	Wires MaintenanceWorker with the new MaintenanceService dependency.
src/factories/maintenance-service-factory.ts	New factory to construct MaintenanceService with EventRepository + settings provider.
src/app/maintenance-worker.ts	Schedules maintenance purge each interval and adds an `isRunning` guard against overlap.
src/@types/settings.ts	Adds typed settings shape for `limits.event.retention.*`.
src/@types/services.ts	Adds `IMaintenanceService`.
src/@types/repositories.ts	Adds retention option and purge-count types and repository method contract.
resources/default-settings.yaml	Adds default (disabled) retention config and default kind whitelist.
CONFIGURATION.md	Documents the new retention settings keys and behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/app/maintenance-worker.ts

  private async onSchedule(): Promise<void> {
    const currentSettings = this.settings()

+    await this.maintenanceService.clearOldEvents()
+
    if (!path(['payments','enabled'], currentSettings)) {
      return


CONFIGURATION.md

+| limits.event.retention.maxDays              | Maximum number of days to retain events. Purge deletes events that are expired (`expires_at`), soft-deleted (`deleted_at`), or older than this window (`created_at`). Any non-positive value disables retention purge. |
+| limits.event.retention.kind.whitelist       | Event kinds excluded from retention purge. NIP-62 `REQUEST_TO_VANISH` is always excluded from retention purge, even if not listed here. |
+| limits.event.retention.pubkey.whitelist     | Public keys excluded from retention purge. |


Justxd22 mentioned this pull request Apr 9, 2026

[REQUEST] Purging old events #359

Open

cameri requested changes Apr 9, 2026

View reviewed changes

resources/default-settings.yaml Outdated Show resolved Hide resolved

CONFIGURATION.md Outdated Show resolved Hide resolved

src/app/maintenance-worker.ts Outdated Show resolved Hide resolved

src/repositories/event-repository.ts Outdated Show resolved Hide resolved

Justxd22 force-pushed the feat/event-retention-purge-opt-in branch 2 times, most recently from 49044d2 to 7173f4d Compare April 9, 2026 16:57

Justxd22 requested a review from cameri April 9, 2026 17:05

cameri reviewed Apr 10, 2026

View reviewed changes

src/repositories/event-repository.ts Outdated Show resolved Hide resolved

cameri reviewed Apr 10, 2026

View reviewed changes

resources/default-settings.yaml Outdated Show resolved Hide resolved

cameri reviewed Apr 10, 2026

View reviewed changes

src/repositories/event-repository.ts Outdated Show resolved Hide resolved

cameri requested review from Copilot April 10, 2026 02:51

Copilot started reviewing on behalf of cameri April 10, 2026 02:52 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

src/app/maintenance-worker.ts Show resolved Hide resolved

src/repositories/event-repository.ts Outdated Show resolved Hide resolved

Copilot AI reviewed Apr 10, 2026

View reviewed changes

src/services/maintenance-service.ts Outdated Show resolved Hide resolved

src/repositories/event-repository.ts Outdated Show resolved Hide resolved

CONFIGURATION.md Outdated Show resolved Hide resolved

Justxd22 force-pushed the feat/event-retention-purge-opt-in branch from 7173f4d to bde7cde Compare April 11, 2026 12:07

Justxd22 requested review from cameri and Copilot April 11, 2026 12:08

Copilot started reviewing on behalf of Justxd22 April 11, 2026 12:09 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

feat: add opt-in event retention purge (cameri#359)

57dad0f

Justxd22 force-pushed the feat/event-retention-purge-opt-in branch from bde7cde to 57dad0f Compare April 11, 2026 14:04

Justxd22 requested a review from Copilot April 11, 2026 14:05

Copilot started reviewing on behalf of Justxd22 April 11, 2026 14:06 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

Conversation

Justxd22 commented Apr 9, 2026

Description

Related Issue

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

cameri commented Apr 9, 2026

Uh oh!

cameri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Justxd22 commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cameri commented Apr 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cameri commented Apr 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants