feat(controller): observable vault scanner skip failures (spec 043)

[bborbe]

Summary

Restores Prometheus observability for vault task files silently skipped by the scanner. Closes the dedup safety net's silent failure mode that hid 24 broken files (4 of them human_review PR-review tasks) across the 2026-05-31 / 2026-06-01 incident.

• New counter agent_controller_vault_scanner_skipped_files_total{reason} with five pre-initialised label values (invalid_frontmatter, duplicate_frontmatter_invalid, empty_status, inject_task_identifier_failed, read_failed); increments exactly once per skipped file per scan cycle.
• Promoted four glog.Warningf("skipping …") sites to glog.Errorf so existing log-scrape alert routing surfaces the event. read_failed and inject_task_identifier_failed remain Warningf (counter alone is sufficient — both are rare, both are operator-actionable via the metric).
• Spec: specs/in-progress/043-surface-vault-scanner-skip-failures.md. Doctrine advanced: [[Make Parked Agent Tasks Visible to Operator]]. Systematic-prevention half of [[Fix unresolved merge conflicts in OpenClaw vault]].

Test plan

• Unit tests green — scanner pkg coverage 88.8%, metrics pkg 68.8%, all 40 specs pass
• make precommit exits 0 (lint funlen suppressed with justification on Run and processFile)
• Dev deploy via BRANCH=dev make buca — image SHA ef52275a… running on agent-task-controller-0, startup log reports version=v0.63.35-19-g3508338 commit=3508338
• Post-merge: redeploy :dev from agent-dev worktree to realign with master; then deploy to prod
• Operator-driven AC#13 — plant a test file with <<<<<<< markers in dev OpenClaw vault, observe invalid_frontmatter counter increment + glog.Errorf log line, then remove

[ben-s-pull-request-reviewer]

PR Review Summary

Branch: feature/vault-scanner-skip-alerts → master
PR: #14

Code Review Findings

Correctness:

• 4 glog.Warningf("skipping...") sites correctly promoted to glog.Errorf (lines 239, 245, 254, 299)
• read_failed (line 229) and inject_task_identifier_failed (line 326) correctly remain at Warningf per spec
• 6 counter increment calls matching 6 skip sites (5 unique reasons, with invalid_frontmatter used at 2 sites)
• Counter pre-initialization in init() for all 5 label values (lines 201-209)
• Constructor signature correctly accepts metrics.Metrics interface

Tests:

• vault_scanner_test.go has comprehensive SkippedFilesTotal counter Context for all 5 reasons (lines 780-1059)
• metrics_test.go verifies all 5 reason labels are pre-initialized (lines 98-110)
• vault_scanner_internal_test.go tests inject_task_identifier_failed counter
• AC#6 invariant test at line 1035 validates skip-site/counter-call parity

Documentation:

• docs/controller-design.md updated with counter documentation (line 28)
• CHANGELOG.md has unreleased entry with incident reference
• Spec correctly references vault doctrine

No issues found.

{
  "verdict": "approve",
  "summary": "Implementation fully satisfies spec-043. The vault scanner now emits a labelled Prometheus counter (agent_controller_vault_scanner_skipped_files_total{reason}) at all 6 skip sites, with 4 operator-actionable log lines promoted from Warningf to Errorf. All 5 reason labels are pre-initialised to zero. Comprehensive Ginkgo test coverage validates per-reason increment behaviour and the skip-site/counter-call invariant.",
  "comments": [],
  "concerns_addressed": [
    "correctness: 4 glog.Warningf skip sites promoted to glog.Errorf — errorf at lines 239,245,254,299; read_failed and inject_task_identifier_failed correctly remain at Warningf per spec",
    "correctness: Counter increment logic is idempotent per scan cycle — hash check prevents double-counting for empty_status; re-process correctly re-increments for broken files",
    "correctness: 5 label values are a closed enum (constants), no high cardinality risk",
    "tests: All 5 skip reasons covered by Ginkgo Context blocks in vault_scanner_test.go and vault_scanner_internal_test.go",
    "performance: Counter pre-initialisation in init() uses Add(0) for 5 label values — negligible memory overhead"
  ]
}