From 1f45229203c0100d0875129f0a22d1a638b754ea Mon Sep 17 00:00:00 2001 From: nick-gorman Date: Wed, 10 Jun 2026 22:05:48 +1000 Subject: [PATCH] Bring network and resource schemas up to the newer schema format The costs, generators and storage schemas were restructured (2963704) with sectioned descriptions (Source tables / Source notes / If absent), per-column absence semantics and numeric value constraints, but the earlier network_ and resource_ schemas were left in the old flat style. This aligns them so all schemas share one format ahead of the validation layer landing. No nan_fill was added anywhere: in these tables a blank cell carries meaning (no physical limit, hard cap, static timeslice) rather than standing in for a default. resource_limits has no new-format templater yet, so its source table is inferred from the legacy REZ build-limits templater. Co-Authored-By: Claude Fable 5 --- .../schemas/network_expansion_options.yaml | 34 +++++++++++++++---- .../validation/schemas/network_geography.yaml | 18 +++++++--- ...ork_transmission_path_expansion_costs.yaml | 26 ++++++++++---- .../network_transmission_path_limits.yaml | 24 +++++++++---- .../schemas/network_transmission_paths.yaml | 21 +++++++++--- .../validation/schemas/resource_limits.yaml | 20 +++++++---- 6 files changed, 110 insertions(+), 33 deletions(-) diff --git a/src/ispypsa/validation/schemas/network_expansion_options.yaml b/src/ispypsa/validation/schemas/network_expansion_options.yaml index c11b8338..dd8d9ac4 100644 --- a/src/ispypsa/validation/schemas/network_expansion_options.yaml +++ b/src/ispypsa/validation/schemas/network_expansion_options.yaml @@ -4,10 +4,25 @@ unique: - [expansion_id, expansion_type] description: > Defines the selected expansion option and allowed capacity increase for - expandable network elements. Covers both physical transmission paths and - group constraints that can be relaxed through expansion. Physical paths - emit two rows (forward and reverse); constraint groups emit one - constraint_relaxation row. + expandable network elements. + + Covers both physical transmission paths and group constraints that can be + relaxed through expansion. Physical paths emit two rows (forward and + reverse); constraint groups emit one constraint_relaxation row. + + Source tables: + - `flow_path_augmentation_options_` + - `rez_augmentation_options_` + - `flow_path_augmentation_costs__` + - `rez_augmentation_costs__` + + Source notes: + The `templater` selects the least-cost augmentation option for each network + element, so the cost tables determine which option's capacity increase + appears here. + + If absent: + No network elements can be expanded by the model. columns: expansion_id: type: string @@ -31,8 +46,15 @@ columns: type: float required: true units: MW - description: Capacity increase provided by the selected expansion option for this expansion_type. + gte: 0.0 + description: > + Capacity increase provided by the selected expansion option for this + expansion_type. expansion_option: type: string required: false - description: Name of the selected augmentation option, retained for traceability. + description: > + Name of the selected augmentation option, retained for traceability. + + If absent (or empty): + No effect on the model; the column is informational only. diff --git a/src/ispypsa/validation/schemas/network_geography.yaml b/src/ispypsa/validation/schemas/network_geography.yaml index 8aa0c6ed..23f79226 100644 --- a/src/ispypsa/validation/schemas/network_geography.yaml +++ b/src/ispypsa/validation/schemas/network_geography.yaml @@ -3,10 +3,15 @@ required: true unique: - [geo_id] description: > - Unified table of all spatial entities in the network model: - NEM subregions, NEM regions, and renewable energy zones (REZs). + Unified table of all spatial entities in the network model. + + Contains NEM subregions, NEM regions, and renewable energy zones (REZs). Subregion and region entries are for spatial rollups and results formatting only. + + Source tables: + - `sub_regional_reference_nodes` + - `renewable_energy_zones` columns: geo_id: type: string @@ -29,6 +34,9 @@ columns: type: string required: false description: > - ISP sub-region the geography belongs to. Only present when - regional_granularity is "sub_regions"; the column is omitted entirely for - "nem_regions" and "single_region". For subregion entries, equals geo_id. + ISP sub-region the geography belongs to. For subregion entries, equals + geo_id. + + If absent: + The template was built with regional_granularity "nem_regions" or + "single_region"; the column is only present for "sub_regions". diff --git a/src/ispypsa/validation/schemas/network_transmission_path_expansion_costs.yaml b/src/ispypsa/validation/schemas/network_transmission_path_expansion_costs.yaml index a1e57cc8..fb0c8711 100644 --- a/src/ispypsa/validation/schemas/network_transmission_path_expansion_costs.yaml +++ b/src/ispypsa/validation/schemas/network_transmission_path_expansion_costs.yaml @@ -3,11 +3,24 @@ required: false unique: - [expansion_id, year] description: > - Time-varying annualised expansion costs for network elements, covering both - physical transmission paths and group constraints. - Merges flow path and REZ transmission expansion costs into a single long-format - table. - If this table is absent, network expansion is treated as free. + Time-varying annualised expansion costs for network elements. + + Covers both physical transmission paths and group constraints, merging flow + path and REZ transmission expansion costs into a single long-format table. + + Source tables: + - `flow_path_augmentation_costs__` + - `rez_augmentation_costs__` + - `flow_path_augmentation_options_` + - `rez_augmentation_options_` + + Source notes: + Costs are taken from the cost trajectory of the least-cost augmentation + option selected by the `templater`; the options tables provide the capacity + increase used to express costs per MW. + + If absent: + Network expansion is treated as free. columns: expansion_id: type: string @@ -16,13 +29,14 @@ columns: - network_expansion_options: expansion_id description: Identifier for the expandable network element. year: - type: integer + type: int required: true description: Financial year the cost applies to. cost: type: float required: true units: $/MW + gte: 0.0 description: > Annualised expansion cost, expressed per MW of the maximum directional capacity: total option cost divided by max(forward, reverse) of the diff --git a/src/ispypsa/validation/schemas/network_transmission_path_limits.yaml b/src/ispypsa/validation/schemas/network_transmission_path_limits.yaml index 0e375470..3806cca6 100644 --- a/src/ispypsa/validation/schemas/network_transmission_path_limits.yaml +++ b/src/ispypsa/validation/schemas/network_transmission_path_limits.yaml @@ -8,9 +8,16 @@ custom_validation: direction combination. description: > Static transmission capacity limits for each path by direction and timeslice. - Sourced from flow_path_transfer_capability and interconnector_transfer_capability - in the IASR workbook. - If this table is absent, no transmission capacity limits are enforced. + + Flow path limits vary by demand condition (timeslice); REZ-to-subregion + connection limits are static. + + Source tables: + - `flow_path_transfer_capability` + - `initial_transmission_limits` + + If absent: + No transmission capacity limits are enforced. columns: path_id: type: string @@ -30,11 +37,16 @@ columns: - timeslices: timeslice_id description: > Demand condition the limit applies to. - If absent, capacity is treated as a static limit applying to all conditions. + + If absent (or empty): + Capacity is treated as a static limit applying to all conditions. capacity: type: float required: false units: MW + gte: 0.0 description: > - Transfer capability limits. - NaN indicates a constraint-modelled path with no explicit physical limit. + Transfer capability limit. + + If absent (or empty): + The path is constraint-modelled with no explicit physical limit. diff --git a/src/ispypsa/validation/schemas/network_transmission_paths.yaml b/src/ispypsa/validation/schemas/network_transmission_paths.yaml index 19c90a60..372175b4 100644 --- a/src/ispypsa/validation/schemas/network_transmission_paths.yaml +++ b/src/ispypsa/validation/schemas/network_transmission_paths.yaml @@ -3,10 +3,23 @@ required: false unique: - [path_id] description: > - Transmission paths connecting geographies in the network model, including - inter-subregional flow paths, regional interconnectors, and REZ-to-subregion - connections. If absent, the model is constructed with no transmission paths. - Geographies with no connecting paths are electrically isolated. + Transmission paths connecting geographies in the network model. + + Includes inter-subregional flow paths, regional interconnectors, and + REZ-to-subregion connections. + + Source tables: + - `flow_path_transfer_capability` + - `renewable_energy_zones` + - `flow_path_augmentation_options_` + + Source notes: + Augmentation options for corridors with no existing path (new parallel + corridors) are appended as zero-capacity paths by the `templater`. + + If absent: + The model is constructed with no transmission paths. Geographies with no + connecting paths are electrically isolated. columns: path_id: type: string diff --git a/src/ispypsa/validation/schemas/resource_limits.yaml b/src/ispypsa/validation/schemas/resource_limits.yaml index 59771853..efd6f521 100644 --- a/src/ispypsa/validation/schemas/resource_limits.yaml +++ b/src/ispypsa/validation/schemas/resource_limits.yaml @@ -4,9 +4,15 @@ unique: - [geo_id, resource_type, limit_type] description: > Resource and land use limits for geographies in the network model. + Each row represents a limit on a specific resource type within a geography. Geographies with no rows in this table have no resource limits applied. - If this table is absent, the model is constructed without resource limits. + + Source tables: + - `initial_build_limits` + + If absent: + The model is constructed without resource limits. columns: geo_id: type: string @@ -30,14 +36,16 @@ columns: type: float required: true units: MW + gte: 0.0 description: Limit value. resource_limit_penalty: type: float required: false units: $/MW + gte: 0.0 description: > - Annualised penalty applied if a limit is exceeded. - NaN means no violation is allowed (hard cap). - 0.0 means there is effectively no limit — generation can exceed the - limit at zero cost. - If this column is absent, all limits are treated as hard caps. + Annualised penalty applied if a limit is exceeded. 0.0 means there is + effectively no limit — generation can exceed the limit at zero cost. + + If absent (or empty): + The limit is treated as a hard cap; no violation is allowed.