UN-3479 [FEAT] SDK subpackage for org-to-org data migration

[chandrasekharan-zipstack]

Summary

New unstract.migration sub-package — moves an org's configured resources (adapters, connectors, tags, prompt-studio projects, workflows, tool instances, workflow endpoints, ETL/TASK pipelines, API deployments) into another org via two admin-issued Platform API keys. Works cloud↔cloud, on-prem↔on-prem, or within the same deployment.

Tracked in UN-3479. Backend companion: Zipstack/unstract#1987.

Architecture

• OrgEndpoint — base URL + org slug + Platform API key + configurable api_path_prefix (api/v1 default).
• PlatformClient — entity-scoped HTTP wrapper; Bearer auth; DRF-OPTIONS-driven writable-field set via get_post_schema(entity_path).
• MigrationContext — bundles source/target clients, options, per-run RemapTable (source UUID → target UUID).
• Phase base + per-entity subclasses; each phase: list → per-id GET → POST (or adopt-by-name).
• walker.remap_uuids — JSON walker that rewrites embedded UUIDs using RemapTable.resolve_any before POST.
• MigrationReport — rich table summary + Source → Target UUID Map for audit.
• click CLI: unstract-migrate migrate --source-url ... --source-org ... --target-url ... --target-org ... with env-var fallback for keys, plus --include / --exclude for phase selection and --dry-run.

Phase order is owned by orchestrator.PHASES — phases never call each other:

adapter → connector → tag → custom_tool → workflow → tool_instance
       → workflow_endpoint → pipeline → api_deployment

Phases

Phase	Notes
Adapter	Skips is_friction_less=True. Encrypted adapter_metadata_b survives via per-id GET.
Connector	Same list→GET→POST pattern. Adopt-by-name idempotent.
Tag	Backend now exposes POST/PATCH/DELETE; SDK creates tags on target.
CustomTool	Composite — tool + profile_managers + prompts + republish to registry. Resolves profile adapters by name (UUIDs differ across orgs).
Workflow	tool_metadata UUID-walk during remap.
ToolInstance	≤1 per workflow; FK-rewrites via remap.
WorkflowEndpoint	Rewrites connector + workflow refs.
Pipeline	ETL + TASK only. DEFAULT (legacy v1) and APP (Streamlit) skipped. Warns when source has >1 active key.
APIDeployment	Adopts by api_name. Same key-warning behaviour. Sequenced after workflow_endpoint (serializer requires configured source+destination endpoints).

Constraints (from design ADRs)

• Secrets carried verbatim through the SDK (same posture as the FE; ADR 0006).
• No local state file (ADR 0007); idempotency = name-based GET against target.
• No UUID preservation; embedded refs rewritten via the walker.
• No DB / schema migrations.

Carry-forward gotchas

1. List endpoints often serve stripped payloads (e.g. AdapterListSerializer omits adapter_metadata_b) — every phase does list → per-id GET before POST.
2. PATH_PREFIX is env-controlled on the backend; --api-prefix flag exposed.
3. AuditSerializer auto-adds creator to shared_users M2M — relevant for any cleanup scripting.
4. Service-account is a real User row (is_service_account=True) + OrganizationMember.
5. Backend auto-provisions one active API key per Pipeline/APIDeployment on POST; key UUIDs are editable=False. SDK does not preserve source key values — operator must rotate. Warning is emitted when source has >1 active key.

Test plan

• 56 unit tests across all phases (happy / dry-run / adopt / abort / extra-keys-warning paths) + remap table + walker + HTTP client + CLI.
• Local end-to-end smoke against docker stack (org_Q4qgjLWIbaJlfSts → org_migration_target) — full pipeline creates on first run, adopts idempotently on re-run.
• Cloud↔cloud smoke against staging before un-drafting.
• On-prem↔on-prem smoke before un-drafting.

Out of scope (separate work)

• shared_to_org / shared_users propagation (resource-ownership semantics).
• Prompt-studio uploaded document blobs (file migration).
• API-key value preservation (would need backend design change).
• Pipeline types DEFAULT (legacy) and APP (Streamlit).

Related

• JIRA: UN-3479
• Backend PR: Zipstack/unstract#1987 (feat/org-migration-platform-api-gaps)
• KB (internal): ~/Documents/Obsidian Vault/zipstuff/org-data-migration/

[greptile-apps]

Greptile Summary

This PR introduces unstract.clone, a new SDK sub-package for migrating all configured resources (adapters, connectors, tags, prompt-studio projects, workflows, tool instances, workflow endpoints, ETL/TASK pipelines, API deployments) from one Unstract org to another via two Platform API keys. It ships a click CLI, a thread-pooled phase orchestrator, a JSON UUID walker, and 56 unit tests.

• All major bugs flagged in previous review rounds have been resolved: per-id GET is in place for every phase, the build_post_payload false/zero drop is fixed, the dry-run adopt path no longer fires export_custom_tool, resolve_any is now safely snapshotted, and unresolvable connector references skip rather than null-patch.
• Two minor quality issues remain: _lookup_tool_name in FilesPhase calls list_custom_tools() once per tool in a sequential loop (N+1, same pattern already fixed in CustomToolPhase), and the --api-prefix CLI flag is applied to both source and target endpoints with no way to specify different prefixes for cross-version deployments.

Confidence Score: 5/5

Safe to merge; all blocking correctness bugs from prior rounds have been addressed and only minor quality improvements remain.

Every phase now follows the list → per-id GET → POST pattern, resolve_any is safely snapshotted, the dry-run adopt path no longer fires real writes, build_post_payload correctly preserves False and 0, and the unresolvable-connector PATCH-null was eliminated. The two remaining findings — an N+1 list_custom_tools() call in FilesPhase and a shared --api-prefix CLI flag — are quality/UX concerns with no correctness impact on the migration data.

src/unstract/clone/phases/files.py — _lookup_tool_name N+1 call; src/unstract/clone/cli.py — shared --api-prefix for source and target.

Important Files Changed

Filename	Overview
src/unstract/clone/phases/base.py	Core phase helpers: build_post_payload fix is correct (explicit is not None and != ""); misleading comment about not in (None, "") semantics.
src/unstract/clone/phases/custom_tool.py	All previously flagged bugs fixed: list_custom_tools() hoisted, dry-run adopt path short-circuits before export_custom_tool, bare return replaced with return result.
src/unstract/clone/phases/workflow_endpoint.py	Unresolvable-connector skip and null connection_type coercion both fixed; patch logic is correct.
src/unstract/clone/phases/pipeline.py	Per-id GET before POST is in place; adopt/create/dry-run paths are all correct; extra-key warning logic is sound.
src/unstract/clone/phases/api_deployment.py	Per-id GET before POST added; adopt-by-name idempotency correct; mirrors pipeline pattern faithfully.
src/unstract/clone/phases/workflow.py	Per-id GET (get_workflow) now called before build/remap on the create path; cascade-skip logic via skipped_custom_tool_registry_ids is correct.
src/unstract/clone/context.py	resolve_any snapshotting fix (list(self._table.values())) correctly prevents RuntimeError: dict changed size under concurrent record calls.
src/unstract/clone/client.py	Clean entity-scoped HTTP wrapper; get_post_schema caching is correct (only called from sequential run() methods, not from parallel workers).
src/unstract/clone/phases/files.py	N+1 list_custom_tools() calls in _lookup_tool_name (called once per tool in the Pass 1 loop); all lock usage in parallel pass is correct; retry/backoff logic is sound.
src/unstract/clone/cli.py	Single --api-prefix applied to both source and target prevents cross-version migrations with different path prefixes; all other CLI wiring is correct.
src/unstract/clone/orchestrator.py	Phase ordering is topologically correct; CloneError abort and duration stamping work correctly; clients closed in finally block.
src/unstract/clone/phases/tool_instance.py	Broken-adapter sentinel detection, metadata PATCH, and remap recording are all correct; dry-run paths are consistent.
src/unstract/clone/walker.py	UUID regex walker is correct and safe; unknown UUIDs pass through untouched.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    CLI["CLI / clone()"] --> A["AdapterPhase\nlist → GET → POST/adopt"]
    CLI --> B["ConnectorPhase\nlist → GET → POST/adopt"]
    CLI --> C["TagPhase\nlist → POST/adopt"]
    A --> D["CustomToolPhase\nexport → import/sync-prompts → registry"]
    B --> D
    D --> E["FilesPhase\nlist DM → download → upload"]
    D --> F["WorkflowPhase\nlist → GET → POST/adopt"]
    A --> F
    B --> F
    F --> G["ToolInstancePhase\ncreate → PATCH metadata"]
    D --> G
    G --> H["WorkflowEndpointPhase\nPATCH connector + config"]
    B --> H
    H --> I["PipelinePhase\nlist → GET → POST/adopt"]
    F --> I
    H --> J["APIDeploymentPhase\nlist → GET → POST/adopt"]
    F --> J
    subgraph remap["RemapTable (shared, thread-safe snapshot)"]
        R1["adapter uuid → uuid"]
        R2["connector uuid → uuid"]
        R3["custom_tool uuid → uuid"]
        R4["prompt_studio_registry uuid → uuid"]
        R5["workflow uuid → uuid"]
        R6["tool_instance uuid → uuid"]
    end
    A -.-> R1
    B -.-> R2
    D -.-> R3
    D -.-> R4
    F -.-> R5
    G -.-> R6

Loading

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
src/unstract/clone/phases/files.py:451-471
**N+1 `list_custom_tools()` calls in file-name lookup**

`_lookup_tool_name` is called once per entry in `tool_remap` (line 81), and every call issues a fresh `GET prompt-studio/` request to list all target tools. For N migrated tools this produces N round-trips that return the same payload. The exact same pattern was hoisted out of the loop in `CustomToolPhase` in a previous review round; the same fix applies here: fetch the list once before the loop and build a `{tool_id: tool_name}` dict that subsequent iterations query locally.

### Issue 2 of 3
src/unstract/clone/cli.py:183-194
**`--api-prefix` is shared between source and target**

Both `OrgEndpoint` objects receive the same `api_prefix` value. A migration between a deployment with `PATH_PREFIX=api/v1` and one with `PATH_PREFIX=api/v2` (or any cross-version setup) has no way to supply different prefixes per side. Consider adding `--source-api-prefix` / `--target-api-prefix` flags (with `--api-prefix` kept as a shorthand that sets both when they would be equal), or at minimum document that both deployments must share the same prefix.

### Issue 3 of 3
src/unstract/clone/phases/base.py:47-49
**Misleading comment about `not in (None, "")` semantics**

The comment states that `0 not in (None, "")` "falsely returns True," but in standard Python that expression returns `True` because `0` is neither equal to `None` nor to `""` — the result is correct, not false. The actual footgun in the old code was a truthiness check (`if v:`) that would silently drop `False` and `0`. The current explicit `is not None and != ""` guards are correct; consider rewriting the comment to say "replaced a truthiness guard that dropped `False`/`0`" so the rationale is unambiguous.

_{Reviews (19): Last reviewed commit: "Merge pull request #16 from Zipstack/ci/..." | Re-trigger Greptile}

[jaseemjaskp]

Automated multi-agent review of the clone subpackage (Code Reviewer, Silent-Failure Hunter, Type-Design, Test Analyzer, Comment Analyzer, Code Simplifier). Findings below are grouped by file; severity is tagged inline (P1 highest). Findings already covered by existing review comments (e.g. the greptile dry-run write on custom_tool.py:180, the _extract_adapter_name / list_custom_tools / base.build_post_payload / workflow_endpoint items) are intentionally omitted as fixed/duplicate. Tests pass (75 green).

[jaseemjaskp]

Second-pass review (post-9391ce3). The prior round's P1/P2/P3s are addressed — verified the dry-run adopt short-circuits, file-row counters, narrowed excepts, session close, and connection_type omission. One new correctness-of-reporting issue remains; everything else I'd add is either already deferred-with-rationale (StrEnum option sets, report dict buckets, six-phase run() duplication) or below the bar.