feat(eval): add --simulation flag to uipath debug#1632
Conversation
Mirrors the --simulation flag from `uipath run` (#1624) onto `uipath debug`, so simulation configs can be passed inline as JSON without needing a simulation.json on disk. When --simulation is omitted, the existing simulation.json fallback is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
can you please add some tests here |
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Screen.Recording.2026-05-19.at.3.54.08.PM.mov |
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Chibionos
left a comment
There was a problem hiding this comment.
PR #1632 Review — --simulation flag on uipath debug
Verdict: ✅ Approve after rebase. Clean mirror of the uipath run --simulation flag (#1624). I checked out the head, ran the suite, and traced the flow.
✅ Verified working
- 18/18 tests pass (
tests/cli/test_debug_simulation.py), including the 3 newTestDebugSimulationFlagtests. ruff check+ruff format --checkclean.- Closure capture of
simulation_configinto the nestedexecute_debug_runtimeis correct. except (ValidationError, ValueError)matchescli_run.pyexactly; theconsole.error()early-exit (NoReturn→SystemExit) correctly prevents reachingMiddlewares.next— and the test'sassert not factory_get.calledconfirms it.- All five
patch(...)targets resolve to real names incli_debug's namespace.
🔵 Before merge
- Branch is
CONFLICTINGand bumps2.10.67 → 2.10.68; rebase and resolve the version-line conflict (note #1674 also bumps the version).
Two minor inline observations below — nothing blocking.
Adversarial + deep-review pass; verified on the PR head.
|
|
||
| [options] | ||
| exclude-newer = "2026-05-17T16:21:47.0725551Z" | ||
| exclude-newer = "0001-01-01T00:00:00Z" # This has no effect and is included for backwards compatibility when using relative exclude-newer values. |
There was a problem hiding this comment.
🔵 exclude-newer flipped to epoch zero
exclude-newer changed from the real timestamp 2026-05-17T... to 0001-01-01T00:00:00Z. This is a uv-version artifact and is harmless, but confirm it came from an intentional uv lock regeneration rather than a stray downgrade of the uv toolchain — it can subtly change dependency resolution on the next lock.
| simulation_config = SimulationConfig.model_validate_json(simulation) | ||
| except (ValidationError, ValueError) as e: | ||
| console.error(f"Invalid --simulation config: {e}") | ||
| return |
There was a problem hiding this comment.
🔵 unreachable return
console.error() is NoReturn (raises SystemExit), so this return never executes. Harmless and consistent with the existing pattern at :138; fine to leave. Flagging only so it's a conscious choice.



Summary
Mirrors the
--simulationflag introduced foruipath runin #1624 ontouipath debug, so a simulation config can be passed inline (same JSON schema assimulation.json) without writing it to disk.--simulationoption + parameter tocli_debug.pySimulationConfig.model_validate_json(usespydantic.ValidationErrorfor surfacing errors)--simulationis provided, builds the mocking context viabuild_mocking_context(simulation_config, agent_model)--simulationis omitted, the existingload_simulation_config()fallback (readssimulation.jsonfrom cwd) is preserved — this is the one behavioral difference fromrun, which has no file-based fallbackTest plan
ruff checkandruff format --checkpass on the modified filedebug --simulation(mirroring theTestRunSimulationclass added totests/cli/test_run.pyin feat(eval): add runtime simulation via --simulation flag #1624) are not included in this PR — happy to add in a follow-up if reviewers want them hereScope notes
This PR only mirrors the CLI flag. The sample (
runtime-simulations-agent) and integration testcase (simulation-testcase) added in #1624 exerciseuipath runand are not duplicated fordebugsince the underlyingbuild_mocking_contextplumbing they cover is shared.🤖 Generated with Claude Code