Skip to content

Commit 65ac20e

Browse files
Aryamanz29claude
Aryamanz29
and
claude
committed
[feat] Add Pkl-generated pyatlan_v9 asset models with search field descriptors and overlay system
- Regenerate 557 asset files from Pkl type definitions with field descriptors (ClassVar placeholders + deferred KeywordField/BooleanField/NumericField init) - Add 82 overlay files (_overlays/) with custom methods (creator, updater, policies) - Add /generate-v9-models Claude Code skill for model regeneration from models repo - Add pyatlan_v9 model generation section to README - Post-sync patches: set[str] fields in asset.py, relationship_attributes in related_entity.py, Process.Attributes backward-compat alias Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 3953628 commit 65ac20e

642 files changed

Lines changed: 200739 additions & 41902 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/generate-v9-models/SKILL.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
---
2+
description: Generate pyatlan_v9 msgspec model files by cloning the models repo and running the Pkl code generator
3+
---
4+
5+
# Generate v9 Models
6+
7+
Generates pyatlan_v9 msgspec model files from Pkl type definitions in the atlanhq/models repo.
8+
9+
## Usage
10+
11+
- `/generate-v9-models` — Clone models@master, generate and sync v9 models
12+
- `/generate-v9-models <branch>` — Clone models@<branch> instead of master
13+
- `/generate-v9-models <branch> test` — Also run tests after sync
14+
- `/generate-v9-models test` — Clone models@master and run tests after sync
15+
16+
## Instructions
17+
18+
### 1. Clone or update the models repo
19+
20+
Parse args to determine the branch (default: `master`) and whether to run tests (args contain "test").
21+
22+
The models repo should be cloned as a sibling directory of this repo (atlan-python):
23+
24+
```bash
25+
# Determine paths
26+
SDK_DIR="$(pwd)" # atlan-python root
27+
MODELS_DIR="$(cd .. && pwd)/models"
28+
BRANCH="master" # override with first non-"test" arg
29+
30+
if [ -d "$MODELS_DIR" ]; then
31+
cd "$MODELS_DIR" && git fetch origin && git checkout "$BRANCH" && git pull origin "$BRANCH"
32+
else
33+
git clone --branch "$BRANCH" --single-branch git@github.com:atlanhq/models.git "$MODELS_DIR"
34+
fi
35+
```
36+
37+
### 2. Run Pkl evaluation
38+
39+
From the models repo root, run the Pkl code generator with SDK mode enabled:
40+
41+
```bash
42+
cd "$MODELS_DIR"
43+
OVERLAYS_PATH="${SDK_DIR}/pyatlan_v9/model/assets/_overlays/"
44+
45+
pkl eval typedefs/*.pkl -m . -p sdk=true \
46+
-p targetOutputDir=gen_v9/pyatlan_v9/model/assets/ \
47+
-p internalPackage=pyatlan_v9.model \
48+
-p sdkOverlaysBasePath="$OVERLAYS_PATH"
49+
```
50+
51+
- `sdkOverlaysBasePath` must be an absolute path — Pkl resolves `read?()` relative to the module file, not CWD
52+
- The `-p sdk=true` flag enables SDK code generation (search field descriptors, overlay injection, etc.)
53+
- Output goes to `models/gen_v9/pyatlan_v9/model/assets/` as a staging area
54+
55+
### 3. Selective sync
56+
57+
Copy generated files to the SDK, **excluding** these files that have manual patches or are hand-written:
58+
59+
| File | Reason |
60+
|------|--------|
61+
| `__init__.py` | Hand-written init with `__all__` |
62+
| `entity.py` | Patched: `_metadata_proxies`, `type_name: Any`, `SaveSemantic` |
63+
| `referenceable.py` | Patched: `InternalKeywordField`, field descriptors, helper exports |
64+
| `atlas_glossary.py` | Patched: GTC anchor-in-attributes handling |
65+
| `atlas_glossary_term.py` | Patched: GTC anchor-in-attributes handling |
66+
| `atlas_glossary_category.py` | Patched: GTC anchor-in-attributes handling |
67+
| `quick_sight_dataset.py` | Patched: `useLocalTypeAsPrefix` field naming |
68+
| `quick_sight_dataset_field.py` | Patched: `useLocalTypeAsPrefix` field naming |
69+
| `quick_sight_folder.py` | Patched: `useLocalTypeAsPrefix` field naming |
70+
| `data_quality_rule.py` | Hand-written, not yet generated correctly |
71+
72+
```bash
73+
rsync -av \
74+
--exclude='__init__.py' \
75+
--exclude='entity.py' \
76+
--exclude='referenceable.py' \
77+
--exclude='atlas_glossary.py' \
78+
--exclude='atlas_glossary_term.py' \
79+
--exclude='atlas_glossary_category.py' \
80+
--exclude='quick_sight_dataset.py' \
81+
--exclude='quick_sight_dataset_field.py' \
82+
--exclude='quick_sight_folder.py' \
83+
--exclude='data_quality_rule.py' \
84+
gen_v9/pyatlan_v9/model/assets/ \
85+
"${SDK_DIR}/pyatlan_v9/model/assets/"
86+
```
87+
88+
16 additional types in pyatlan_v9 (persona.py, purpose.py, badge.py, access_control.py, etc.) are hand-written and NOT generated by Pkl — rsync won't touch them since they don't exist in gen_v9.
89+
90+
### 4. Post-sync patches
91+
92+
**a) asset.py** — replace `list[str]` with `set[str]` for 7 fields (in both `Asset` and `AssetAttributes` classes):
93+
94+
```bash
95+
cd "${SDK_DIR}"
96+
sed -i '' \
97+
's/\(owner_users: \)list\[str\]/\1set[str]/g;
98+
s/\(owner_groups: \)list\[str\]/\1set[str]/g;
99+
s/\(admin_users: \)list\[str\]/\1set[str]/g;
100+
s/\(admin_groups: \)list\[str\]/\1set[str]/g;
101+
s/\(viewer_users: \)list\[str\]/\1set[str]/g;
102+
s/\(viewer_groups: \)list\[str\]/\1set[str]/g;
103+
s/\(admin_roles: \)list\[str\]/\1set[str]/g' \
104+
pyatlan_v9/model/assets/asset.py
105+
```
106+
107+
**b) related_entity.py** — ensure `relationship_attributes` field exists after `unique_attributes`. If the generated file doesn't have it, add:
108+
```python
109+
# Relationship-specific attributes
110+
relationship_attributes: Union[dict[str, Any], None, UnsetType] = UNSET
111+
"""Attributes of the relationship itself (e.g., description, status, etc.)."""
112+
```
113+
114+
**c) process.py** — append backward-compat alias at end of file (after deferred field descriptors):
115+
```python
116+
Process.Attributes = Process # backward-compat alias for Process.Attributes.generate_qualified_name
117+
```
118+
119+
### 5. Run tests (if args contain "test")
120+
121+
```bash
122+
cd "${SDK_DIR}" && python -m pytest tests_v9/unit/ -x -q
123+
```
124+
125+
### 6. Report summary
126+
127+
Report: how many files were generated, how many synced, how many excluded, and test results if applicable.
128+
129+
## Notes
130+
131+
- The models repo is cloned from `git@github.com:atlanhq/models.git`
132+
- If `../models` already exists, it fetches and checks out the requested branch instead of re-cloning
133+
- Generated files go to `models/gen_v9/` as a staging area, then are selectively synced
134+
- The `set[str]` patch is needed because Pkl's `multiValued=true` generates `list[str]` but the SDK semantically needs `set[str]` for user/group/role fields
135+
- Overlay files (custom methods like `creator()`, `updater()`, policy helpers) live at `pyatlan_v9/model/assets/_overlays/` in this repo

‎README.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,44 @@ This will:
218218
- 🎨 Format code automatically
219219
- ⚡ Support incremental updates
220220

221+
## 🏗️ pyatlan_v9 Model Generation (msgspec)
222+
223+
The `pyatlan_v9` package uses [msgspec](https://jcristharris.com/msgspec/) `Struct`-based models generated from Pkl type definitions in the [atlanhq/models](https://github.com/atlanhq/models) repo.
224+
225+
### Using Claude Code
226+
227+
The recommended way to regenerate models is via the Claude Code skill:
228+
229+
```bash
230+
# From the atlan-python repo root:
231+
/generate-v9-models # Generate from models@master
232+
/generate-v9-models <branch> # Generate from a specific models branch
233+
/generate-v9-models test # Generate and run tests
234+
/generate-v9-models <branch> test
235+
```
236+
237+
The skill will:
238+
1. Clone/update `atlanhq/models` at `../models`
239+
2. Run the Pkl code generator with SDK mode (`pkl eval typedefs/*.pkl -m . -p sdk=true`)
240+
3. Selectively sync generated files to `pyatlan_v9/model/assets/` (excluding hand-written types)
241+
4. Apply post-sync patches (e.g., `set[str]` fields in `asset.py`)
242+
5. Optionally run `tests_v9/unit/` tests
243+
244+
### Overlay Files
245+
246+
Custom methods (`creator()`, `updater()`, policy helpers, etc.) live in `pyatlan_v9/model/assets/_overlays/`. These are Python files read by the Pkl renderer and injected into generated classes. Each overlay file uses import directives:
247+
248+
- `# IMPORT:` — external imports (not remapped)
249+
- `# INTERNAL_IMPORT:` — internal imports (remapped to `pyatlan_v9.*`)
250+
- `# STDLIB_IMPORT:` — standard library imports
251+
252+
### Hand-written Types
253+
254+
Some types are not yet fully generated and are maintained by hand:
255+
- Infrastructure: `__init__.py`, `entity.py`, `referenceable.py`
256+
- GTC types: `atlas_glossary.py`, `atlas_glossary_term.py`, `atlas_glossary_category.py`
257+
- Others: `persona.py`, `purpose.py`, `badge.py`, `access_control.py`, `auth_policy.py`, etc.
258+
221259
## 📁 Project Structure
222260

223261
Understanding the codebase layout will help you navigate and contribute effectively:

‎pyatlan_v9/model/assets/_init_adf.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,6 @@
88
This module provides convenient imports for all ADF types and their Related variants.
99
"""
1010

11-
from .adf import ADF
12-
from .adf_activity import AdfActivity
13-
from .adf_dataflow import AdfDataflow
14-
from .adf_dataset import AdfDataset
15-
from .adf_linkedservice import AdfLinkedservice
16-
from .adf_pipeline import AdfPipeline
1711
from .adf_related import (
1812
RelatedADF,
1913
RelatedAdfActivity,
@@ -22,6 +16,12 @@
2216
RelatedAdfLinkedservice,
2317
RelatedAdfPipeline,
2418
)
19+
from .adf import ADF
20+
from .adf_activity import AdfActivity
21+
from .adf_dataflow import AdfDataflow
22+
from .adf_dataset import AdfDataset
23+
from .adf_linkedservice import AdfLinkedservice
24+
from .adf_pipeline import AdfPipeline
2525

2626
__all__ = [
2727
"ADF",

‎pyatlan_v9/model/assets/_init_adls.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,16 @@
88
This module provides convenient imports for all ADLS types and their Related variants.
99
"""
1010

11-
from .adls import ADLS
12-
from .adls_account import ADLSAccount
13-
from .adls_container import ADLSContainer
14-
from .adls_object import ADLSObject
1511
from .adls_related import (
1612
RelatedADLS,
1713
RelatedADLSAccount,
1814
RelatedADLSContainer,
1915
RelatedADLSObject,
2016
)
17+
from .adls import ADLS
18+
from .adls_account import ADLSAccount
19+
from .adls_container import ADLSContainer
20+
from .adls_object import ADLSObject
2121

2222
__all__ = [
2323
"ADLS",

‎pyatlan_v9/model/assets/_init_ai.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,16 @@
88
This module provides convenient imports for all AI types and their Related variants.
99
"""
1010

11-
from .ai import AI
12-
from .ai_application import AIApplication
13-
from .ai_model import AIModel
14-
from .ai_model_version import AIModelVersion
1511
from .ai_related import (
1612
RelatedAI,
1713
RelatedAIApplication,
1814
RelatedAIModel,
1915
RelatedAIModelVersion,
2016
)
17+
from .ai import AI
18+
from .ai_application import AIApplication
19+
from .ai_model import AIModel
20+
from .ai_model_version import AIModelVersion
2121

2222
__all__ = [
2323
"AI",

‎pyatlan_v9/model/assets/_init_airflow.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,13 @@
88
This module provides convenient imports for all Airflow types and their Related variants.
99
"""
1010

11+
from .airflow_related import (
12+
RelatedAirflow,
13+
RelatedAirflowDag,
14+
RelatedAirflowTask,
15+
)
1116
from .airflow import Airflow
1217
from .airflow_dag import AirflowDag
13-
from .airflow_related import RelatedAirflow, RelatedAirflowDag, RelatedAirflowTask
1418
from .airflow_task import AirflowTask
1519

1620
__all__ = [

‎pyatlan_v9/model/assets/_init_anaplan.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,6 @@
88
This module provides convenient imports for all Anaplan types and their Related variants.
99
"""
1010

11-
from .anaplan import Anaplan
12-
from .anaplan_app import AnaplanApp
13-
from .anaplan_dimension import AnaplanDimension
14-
from .anaplan_line_item import AnaplanLineItem
15-
from .anaplan_list import AnaplanList
16-
from .anaplan_model import AnaplanModel
17-
from .anaplan_module import AnaplanModule
18-
from .anaplan_page import AnaplanPage
1911
from .anaplan_related import (
2012
RelatedAnaplan,
2113
RelatedAnaplanApp,
@@ -29,6 +21,14 @@
2921
RelatedAnaplanView,
3022
RelatedAnaplanWorkspace,
3123
)
24+
from .anaplan import Anaplan
25+
from .anaplan_app import AnaplanApp
26+
from .anaplan_dimension import AnaplanDimension
27+
from .anaplan_line_item import AnaplanLineItem
28+
from .anaplan_list import AnaplanList
29+
from .anaplan_model import AnaplanModel
30+
from .anaplan_module import AnaplanModule
31+
from .anaplan_page import AnaplanPage
3232
from .anaplan_system_dimension import AnaplanSystemDimension
3333
from .anaplan_view import AnaplanView
3434
from .anaplan_workspace import AnaplanWorkspace

‎pyatlan_v9/model/assets/_init_anomalo.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,16 @@
88
This module provides convenient imports for all Anomalo types and their Related variants.
99
"""
1010

11+
from .anomalo_related import (
12+
RelatedAnomalo,
13+
RelatedAnomaloCheck,
14+
)
1115
from .anomalo import Anomalo
1216
from .anomalo_check import AnomaloCheck
13-
from .anomalo_related import RelatedAnomalo, RelatedAnomaloCheck
1417

15-
__all__ = ["Anomalo", "AnomaloCheck", "RelatedAnomalo", "RelatedAnomaloCheck"]
18+
__all__ = [
19+
"Anomalo",
20+
"AnomaloCheck",
21+
"RelatedAnomalo",
22+
"RelatedAnomaloCheck",
23+
]

‎pyatlan_v9/model/assets/_init_api.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,6 @@
88
This module provides convenient imports for all API types and their Related variants.
99
"""
1010

11-
from .api import API
12-
from .api_field import APIField
13-
from .api_object import APIObject
14-
from .api_path import APIPath
15-
from .api_query import APIQuery
1611
from .api_related import (
1712
RelatedAPI,
1813
RelatedAPIField,
@@ -21,6 +16,11 @@
2116
RelatedAPIQuery,
2217
RelatedAPISpec,
2318
)
19+
from .api import API
20+
from .api_field import APIField
21+
from .api_object import APIObject
22+
from .api_path import APIPath
23+
from .api_query import APIQuery
2424
from .api_spec import APISpec
2525

2626
__all__ = [

‎pyatlan_v9/model/assets/_init_app.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,12 @@
88
This module provides convenient imports for all App types and their Related variants.
99
"""
1010

11+
from .app_related import (
12+
RelatedApp,
13+
RelatedApplication,
14+
RelatedApplicationField,
15+
)
1116
from .app import App
12-
from .app_related import RelatedApp, RelatedApplication, RelatedApplicationField
1317
from .application import Application
1418
from .application_field import ApplicationField
1519

0 commit comments

Comments
 (0)