[feat] Regenerate pyatlan_v9 models with updated Pkl generator

- Regenerate all asset models using updated Pkl generator:
  - set[str] fields now generated natively (no post-sync sed patch)
  - validate/minimize/relate methods removed (dead code)
  - relationship_attributes field in RelatedEntity base class
- Move overlay files to SDK repo (_overlays/ directory)
- Fix overlay imports: ai_model.py, collection.py, connection.py
- Add ruff exclusion for _overlays/ and F821 per-file-ignores
- Update generate-v9-models skill: sdkOnly mode, temp staging,
  ruff auto-fix+format step, removed asset.py sed patch
- Update process_test.py to use Process.generate_qualified_name

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Aryamanz29

.claude/skills/generate-v9-models/SKILL.md

@@ -36,25 +36,28 @@ fi
 
 ### 2. Run Pkl evaluation
 
-From the models repo root, run the Pkl code generator with SDK mode enabled:
+From the models repo root, run the Pkl code generator in **SDK-only mode** (`-p sdkOnly=true`). This generates only Python SDK files — no JSON typedefs, no frontend code, no samples.
+
+Use a temp directory for staging so no generated files land in the models repo:
 
 ```bash
 cd "$MODELS_DIR"
+STAGING_DIR="$(mktemp -d)"
 OVERLAYS_PATH="${SDK_DIR}/pyatlan_v9/model/assets/_overlays/"
 
-pkl eval typedefs/*.pkl -m . -p sdk=true \
-  -p targetOutputDir=gen_v9/pyatlan_v9/model/assets/ \
+pkl eval typedefs/*.pkl -m "$STAGING_DIR" -p sdkOnly=true -p sdk=true \
+  -p targetOutputDir=pyatlan_v9/model/assets/ \
   -p internalPackage=pyatlan_v9.model \
   -p sdkOverlaysBasePath="$OVERLAYS_PATH"
 ```
 
+- `-p sdkOnly=true` skips JSON typedef generation — only Python SDK files are produced
 - `sdkOverlaysBasePath` must be an absolute path — Pkl resolves `read?()` relative to the module file, not CWD
-- The `-p sdk=true` flag enables SDK code generation (search field descriptors, overlay injection, etc.)
-- Output goes to `models/gen_v9/pyatlan_v9/model/assets/` as a staging area
+- Output goes to a temp staging directory (nothing written to models repo)
 
 ### 3. Selective sync
 
-Copy generated files to the SDK, **excluding** these files that have manual patches or are hand-written:
+Copy generated files from the staging dir to the SDK, **excluding** these files that have manual patches or are hand-written:
 
 | File | Reason |
 |------|--------|
@@ -81,55 +84,49 @@ rsync -av \
   --exclude='quick_sight_dataset_field.py' \
   --exclude='quick_sight_folder.py' \
   --exclude='data_quality_rule.py' \
-  gen_v9/pyatlan_v9/model/assets/ \
+  "${STAGING_DIR}/pyatlan_v9/model/assets/" \
   "${SDK_DIR}/pyatlan_v9/model/assets/"
+
+# Clean up staging dir
+rm -rf "$STAGING_DIR"
 ```
 
-16 additional types in pyatlan_v9 (persona.py, purpose.py, badge.py, access_control.py, etc.) are hand-written and NOT generated by Pkl — rsync won't touch them since they don't exist in gen_v9.
+16 additional types in pyatlan_v9 (persona.py, purpose.py, badge.py, access_control.py, etc.) are hand-written and NOT generated by Pkl — rsync won't touch them since they don't exist in staging.
 
 ### 4. Post-sync patches
 
-**a) asset.py** — replace `list[str]` with `set[str]` for 7 fields (in both `Asset` and `AssetAttributes` classes):
-
-```bash
-cd "${SDK_DIR}"
-sed -i '' \
-  's/\(owner_users: \)list\[str\]/\1set[str]/g;
-   s/\(owner_groups: \)list\[str\]/\1set[str]/g;
-   s/\(admin_users: \)list\[str\]/\1set[str]/g;
-   s/\(admin_groups: \)list\[str\]/\1set[str]/g;
-   s/\(viewer_users: \)list\[str\]/\1set[str]/g;
-   s/\(viewer_groups: \)list\[str\]/\1set[str]/g;
-   s/\(admin_roles: \)list\[str\]/\1set[str]/g' \
-  pyatlan_v9/model/assets/asset.py
-```
-
-**b) related_entity.py** — ensure `relationship_attributes` field exists after `unique_attributes`. If the generated file doesn't have it, add:
+**related_entity.py** — ensure `relationship_attributes` field exists after `unique_attributes`. This file is not generated in sdkOnly mode, so it persists across regens. If starting from scratch, add:
 ```python
     # Relationship-specific attributes
     relationship_attributes: Union[dict[str, Any], None, UnsetType] = UNSET
     """Attributes of the relationship itself (e.g., description, status, etc.)."""
 ```
 
-**c) process.py** — append backward-compat alias at end of file (after deferred field descriptors):
-```python
-Process.Attributes = Process  # backward-compat alias for Process.Attributes.generate_qualified_name
+### 5. Run ruff auto-fix and format
+
+After syncing and patching, run ruff to fix unused imports and format the generated files:
+
+```bash
+cd "${SDK_DIR}"
+uv run ruff check --fix --select F401,F811 pyatlan_v9/
+uv run ruff format pyatlan_v9/
 ```
 
-### 5. Run tests (if args contain "test")
+### 6. Run tests (if args contain "test")
 
 ```bash
 cd "${SDK_DIR}" && python -m pytest tests_v9/unit/ -x -q
 ```
 
-### 6. Report summary
+### 7. Report summary
 
 Report: how many files were generated, how many synced, how many excluded, and test results if applicable.
 
 ## Notes
 
 - The models repo is cloned from `git@github.com:atlanhq/models.git`
 - If `../models` already exists, it fetches and checks out the requested branch instead of re-cloning
-- Generated files go to `models/gen_v9/` as a staging area, then are selectively synced
-- The `set[str]` patch is needed because Pkl's `multiValued=true` generates `list[str]` but the SDK semantically needs `set[str]` for user/group/role fields
+- `-p sdkOnly=true` ensures only Python SDK files are generated (no JSON typedefs written to models repo)
+- Generated files go to a temp staging dir, then are selectively synced to `atlan-python/pyatlan_v9/model/assets/`
+- Fields with `useSetType=true` in the Pkl typedefs generate `set[str]` instead of `list[str]` (used for user/group/role fields)
 - Overlay files (custom methods like `creator()`, `updater()`, policy helpers) live at `pyatlan_v9/model/assets/_overlays/` in this repo

pyatlan_v9/model/assets/_init_orchestration.py

@@ -8,7 +8,4 @@
 This module provides convenient imports for all Orchestration types and their Related variants.
 """
 
-
-__all__ = [
-
-]
+__all__ = []

pyatlan_v9/model/assets/_init_sage_maker_unified_studio.py

@@ -20,8 +20,12 @@
 from .sage_maker_unified_studio_asset import SageMakerUnifiedStudioAsset
 from .sage_maker_unified_studio_asset_schema import SageMakerUnifiedStudioAssetSchema
 from .sage_maker_unified_studio_project import SageMakerUnifiedStudioProject
-from .sage_maker_unified_studio_published_asset import SageMakerUnifiedStudioPublishedAsset
-from .sage_maker_unified_studio_subscribed_asset import SageMakerUnifiedStudioSubscribedAsset
+from .sage_maker_unified_studio_published_asset import (
+    SageMakerUnifiedStudioPublishedAsset,
+)
+from .sage_maker_unified_studio_subscribed_asset import (
+    SageMakerUnifiedStudioSubscribedAsset,
+)
 
 __all__ = [
     "RelatedSageMakerUnifiedStudio",

pyatlan_v9/model/assets/_overlays/ai_model.py

@@ -1,6 +1,9 @@
+# STDLIB_IMPORT: from typing import Dict, List
 # IMPORT: from pyatlan.model.enums import AIDatasetType, AtlanConnectorType
 # IMPORT: from pyatlan.utils import to_camel_case
 # INTERNAL_IMPORT: from pyatlan.utils import init_guid, validate_required_fields
+# INTERNAL_IMPORT: from pyatlan.model.transform import get_type
+# INTERNAL_IMPORT: from pyatlan.model.assets.process import Process
 
     @classmethod
     @init_guid

pyatlan_v9/model/assets/_overlays/collection.py

@@ -1,3 +1,5 @@
+# STDLIB_IMPORT: from typing import TYPE_CHECKING
+# STDLIB_IMPORT: from uuid import uuid4
 # IMPORT: from pyatlan.errors import AtlanError
 # IMPORT: from pyatlan.errors import ErrorCode
 # INTERNAL_IMPORT: from pyatlan.utils import init_guid, validate_required_fields

pyatlan_v9/model/assets/_overlays/connection.py

@@ -1,3 +1,4 @@
+# STDLIB_IMPORT: from typing import TYPE_CHECKING, List, Optional
 # IMPORT: from pyatlan.model.enums import AtlanConnectorType
 # INTERNAL_IMPORT: from pyatlan.utils import init_guid, validate_required_fields
 

pyatlan_v9/model/assets/adf.py

@@ -16,7 +16,6 @@
 
 from typing import Any, ClassVar, Union
 
-import msgspec
 from msgspec import UNSET, UnsetType
 
 from .airflow_related import RelatedAirflowTask
@@ -43,16 +42,18 @@
 from .schema_registry_related import RelatedSchemaRegistrySubject
 from .soda_related import RelatedSodaCheck
 from .spark_related import RelatedSparkJob
-from pyatlan_v9.model.conversion_utils import categorize_relationships, merge_relationships
+from pyatlan_v9.model.conversion_utils import (
+    categorize_relationships,
+    merge_relationships,
+)
 from pyatlan_v9.model.serde import Serde, get_serde
 from pyatlan_v9.model.transform import register_asset
 
-from .adf_related import RelatedADF
-
 # =============================================================================
 # FLAT ASSET CLASS
 # =============================================================================
 
+
 @register_asset
 class ADF(Asset):
     """
@@ -170,7 +171,9 @@ class ADF(Asset):
     readme: RelatedReadme | None | UnsetType = UNSET
     """README that is linked to this asset."""
 
-    schema_registry_subjects: list[RelatedSchemaRegistrySubject] | None | UnsetType = UNSET
+    schema_registry_subjects: list[RelatedSchemaRegistrySubject] | None | UnsetType = (
+        UNSET
+    )
     """"""
 
     soda_checks: list[RelatedSodaCheck] | None | UnsetType = UNSET
@@ -185,30 +188,6 @@ class ADF(Asset):
     def __post_init__(self) -> None:
         self.type_name = "ADF"
 
-    # =========================================================================
-    # SDK Methods
-    # =========================================================================
-
-    def validate(self, for_creation: bool = False) -> None:
-        errors: list[str] = []
-        if self.type_name is UNSET:
-            errors.append("type_name is required")
-        if self.name is UNSET:
-            errors.append("name is required")
-        if self.qualified_name is UNSET or self.qualified_name is None:
-            errors.append("qualified_name is required")
-        if errors:
-            raise ValueError(f"ADF validation failed: {errors}")
-
-    def minimize(self) -> "ADF":
-        self.validate()
-        return ADF(qualified_name=self.qualified_name, name=self.name)
-
-    def relate(self) -> "RelatedADF":
-        if self.guid is not UNSET:
-            return RelatedADF(guid=self.guid)
-        return RelatedADF(qualified_name=self.qualified_name)
-
     # =========================================================================
     # Optimized Serialization Methods (override Asset base class)
     # =========================================================================
@@ -260,6 +239,7 @@ def from_json(json_data: str | bytes, serde: Serde | None = None) -> ADF:
 # NESTED FORMAT CLASSES
 # =============================================================================
 
+
 class ADFAttributes(AssetAttributes):
     """ADF-specific attributes for nested API format."""
 
@@ -269,6 +249,7 @@ class ADFAttributes(AssetAttributes):
     adf_asset_folder_path: str | None | UnsetType = UNSET
     """Defines the folder path in which this ADF asset exists."""
 
+
 class ADFRelationshipAttributes(AssetRelationshipAttributes):
     """ADF-specific relationship attributes for nested API format."""
 
@@ -344,7 +325,9 @@ class ADFRelationshipAttributes(AssetRelationshipAttributes):
     readme: RelatedReadme | None | UnsetType = UNSET
     """README that is linked to this asset."""
 
-    schema_registry_subjects: list[RelatedSchemaRegistrySubject] | None | UnsetType = UNSET
+    schema_registry_subjects: list[RelatedSchemaRegistrySubject] | None | UnsetType = (
+        UNSET
+    )
     """"""
 
     soda_checks: list[RelatedSodaCheck] | None | UnsetType = UNSET
@@ -356,6 +339,7 @@ class ADFRelationshipAttributes(AssetRelationshipAttributes):
     output_from_spark_jobs: list[RelatedSparkJob] | None | UnsetType = UNSET
     """"""
 
+
 class ADFNested(AssetNested):
     """ADF in nested API format for high-performance serialization."""
 
@@ -364,6 +348,7 @@ class ADFNested(AssetNested):
     append_relationship_attributes: ADFRelationshipAttributes | UnsetType = UNSET
     remove_relationship_attributes: ADFRelationshipAttributes | UnsetType = UNSET
 
+
 # =============================================================================
 # CONVERSION HELPERS & CONSTANTS
 # =============================================================================
@@ -400,19 +385,22 @@ class ADFNested(AssetNested):
     "output_from_spark_jobs",
 ]
 
+
 def _populate_adf_attrs(attrs: ADFAttributes, obj: ADF) -> None:
     """Populate ADF-specific attributes on the attrs struct."""
     _populate_asset_attrs(attrs, obj)
     attrs.adf_factory_name = obj.adf_factory_name
     attrs.adf_asset_folder_path = obj.adf_asset_folder_path
 
+
 def _extract_adf_attrs(attrs: ADFAttributes) -> dict:
     """Extract all ADF attributes from the attrs struct into a flat dict."""
     result = _extract_asset_attrs(attrs)
     result["adf_factory_name"] = attrs.adf_factory_name
     result["adf_asset_folder_path"] = attrs.adf_asset_folder_path
     return result
 
+
 # =============================================================================
 # CONVERSION FUNCTIONS
 # =============================================================================
@@ -452,6 +440,7 @@ def _adf_to_nested(adf: ADF) -> ADFNested:
         remove_relationship_attributes=remove_rels,
     )
 
+
 def _adf_from_nested(nested: ADFNested) -> ADF:
     """Convert nested format to flat ADF."""
     attrs = nested.attributes if nested.attributes is not UNSET else ADFAttributes()
@@ -461,7 +450,7 @@ def _adf_from_nested(nested: ADFNested) -> ADF:
         nested.append_relationship_attributes,
         nested.remove_relationship_attributes,
         _ADF_REL_FIELDS,
-        ADFRelationshipAttributes
+        ADFRelationshipAttributes,
     )
     return ADF(
         guid=nested.guid,
@@ -488,6 +477,7 @@ def _adf_from_nested(nested: ADFNested) -> ADF:
         **merged_rels,
     )
 
+
 def _adf_to_nested_bytes(adf: ADF, serde: Serde) -> bytes:
     """Convert flat ADF to nested JSON bytes."""
     return serde.encode(_adf_to_nested(adf))
@@ -498,6 +488,7 @@ def _adf_from_nested_bytes(data: bytes, serde: Serde) -> ADF:
     nested = serde.decode(data, ADFNested)
     return _adf_from_nested(nested)
 
+
 # ---------------------------------------------------------------------------
 # Deferred field descriptor initialization
 # ---------------------------------------------------------------------------
@@ -535,4 +526,4 @@ def _adf_from_nested_bytes(data: bytes, serde: Serde) -> ADF:
 ADF.SCHEMA_REGISTRY_SUBJECTS = RelationField("schemaRegistrySubjects")
 ADF.SODA_CHECKS = RelationField("sodaChecks")
 ADF.INPUT_TO_SPARK_JOBS = RelationField("inputToSparkJobs")
-ADF.OUTPUT_FROM_SPARK_JOBS = RelationField("outputFromSparkJobs")
+ADF.OUTPUT_FROM_SPARK_JOBS = RelationField("outputFromSparkJobs")