Client-orchestrated voice agent: streaming TTS, --files sandbox, spoken approval

[alexkroman]

Summary

Builds out assembly agent-cascade (a.k.a. the live voice agent): the same live terminal conversation as assembly agent, but client-orchestrated — the engine wires Streaming STT → the LLM Gateway → streaming TTS itself (via a deepagents reply brain) instead of talking to the Voice Agent endpoint. Sandbox-only, since streaming TTS has no prod host.

Highlights across the branch:

• Streaming reply leg (brain.build_streamer): tokens stream from a deepagents graph, are buffered into clauses (pop_clauses), and each clause is synthesized with streaming TTS so audio starts on the first frame instead of after the whole reply.
• Voice-only Textual TUI (LiveAgentApp): transcript + animated voice bar (listening/thinking/speaking), no text input; falls back to plain line output for file/sample/--json/non-TTY.
• --files (off by default): swaps in a real-cwd SandboxedShellBackend with OS-sandboxed execute (sandbox-exec on macOS, bwrap on Linux, refused elsewhere — never an unconfined fallback), write/edit/execute gated through an approval modal, plus durable per-project memory.
• M2 — a gated, gateway-bound general-purpose subagent (deepagents' task tool); delegated writes surface at the parent approval gate.
• M3 — hands-free spoken approval: an open approval modal can be resolved by voice (fail-safe to reject; destructive commands still require a keypress).
• Tools: keyless Open-Meteo weather, read-a-URL (web + PDF), and local date/time.
• Defaults streaming/live/batch to universal-3-5-pro; removes the assembly code command and relocates its shared modules into agent_cascade/.

105 commits, 142 files (+15,190 / −9,502).

This session's commit

The final commit (test(live): split brain tests…) brings the tree green under scripts/check.sh, which surfaced a chain of gate failures once the full clone established the real merge-base:

• Split test_agent_cascade_brain.py (was 521 lines) under the 500-line gate; extracted the write-approval tests to a sibling file (added to the pyright tests ignore list, like the other deepagents-boundary brain tests).
• Added coverage for submit_voice_approval (kept patch coverage at 100%).
• Killed two mutation survivors (modals._expanded default; engine._awaiting_approval init=).
• Fixed two escape-hatch-gate false positives: a comment quoting no cover pragma text, and the cast\( pattern matching the new _forecast() weather function (→ \bcast\().

./scripts/check.sh prints All checks passed.

⚠️ Merge conflict with main

This branch conflicts with main — recent merges (#262 making --files default-on, #264 delegating context-window management to deepagents' SummarizationMiddleware, plus #261) touched the same agent_cascade/ files (notably engine.py) and the docs. The conflicts will need resolving before this can land; I left origin/main unmerged rather than guess at the resolution. Happy to do that as a follow-up if you'd like.

🤖 Generated with Claude Code

---

Generated by Claude Code

[aikido-pr-checks]

_generate_reply no longer unconditionally clears _awaiting_approval on exit; a failure/timeout during an approval pause can leave the gate stuck and misroute later turns as approval input.

Details

✨ AI Reasoning
​​1) The reply loop can enter an approval-pause state that sets a gate flag.
​2) The changed control flow removed unconditional cleanup that previously reset that flag after any exit path.
​3) Early returns on timeout/failure now bypass a guaranteed clear.
​4) That can leave the session in a stale approval-wait state, causing subsequent finalized turns to be misrouted instead of answered.
​5) This is a direct control-flow bug introduced by the refactor, not a pre-existing condition.

🔧 How do I fix it?
Trace execution paths carefully. Ensure precondition checks happen before using values, validate ranges before checking impossible conditions, and don't check for states that the code has already ruled out.

_{Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info}

[aikido-pr-checks]

_is_boundary now treats end-of-buffer as a boundary, so pop_clauses can flush punctuation at partial chunk ends (e.g., decimals split across deltas), producing incorrect clause segmentation during streaming.

Details

✨ AI Reasoning
​​1) The code is trying to split streamed reply text into speakable clauses without breaking mid-token.
​2) The new condition marks end-of-buffer as a boundary.
​3) In incremental streaming, buffers frequently end at temporary token boundaries, so punctuation at chunk end is not guaranteed to be final.
​4) That makes premature clause emission possible, producing incorrect spoken output and transcript chunking.
​5) This is not a stylistic preference; it changes runtime behavior in a way that can produce wrong results.

🔧 How do I fix it?
Trace execution paths carefully. Ensure precondition checks happen before using values, validate ranges before checking impossible conditions, and don't check for states that the code has already ruled out.

_{Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info}

[aikido-pr-checks]

Replacing list accumulation with 'buffer + item.text' causes repeated string allocations in the streaming reply loop; use list append + single join to avoid O(n^2) behavior.

Details

✨ AI Reasoning
​This change is in the reply-leg streaming path that processes many incremental SpeechDelta objects. Previously, when a tool had been used the code accumulated pieces by appending to a list (O(1) per append) and joined once at the end. The new code returns buffer + item.text when used_tool is true, which performs a new string allocation/copy per delta, making the cost O(n^2) in the number/size of deltas. Because this runs while streaming replies (many small deltas), the string-concat-in-loop pattern creates repeated allocations and copying that will noticeably increase CPU and memory usage for longer replies. The fix is straightforward: revert to collecting fragments in a list and join once (or use an io.StringIO or list-append+join strategy).

🔧 How do I fix it?
Move constant work outside loops. Use StringBuilder instead of string concatenation in loops. Cache compiled regex patterns. Use hash-based lookups instead of nested loops. Batch database operations instead of N+1 queries.

_{Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info}