Bring the translator and pypsa_build up to the new table formats#117
Open
nick-gorman wants to merge 8 commits into
Open
Bring the translator and pypsa_build up to the new table formats#117nick-gorman wants to merge 8 commits into
nick-gorman wants to merge 8 commits into
Conversation
The templater has emitted custom_constraints, custom_constraints_lhs and custom_constraints_rhs since the PLEXOS templater landed, but the tables had no schemas, so user-supplied versions of them had no documented contract. The schemas pin the vocabulary the translator now relies on: direction senses (<=, >=, =), term types (including load, which the 7.5 PLEXOS extract emits for Node Load Coefficient terms), the timeslice cross-reference, and date_from semantics (the value active at the start of an investment period applies for that whole period). Also fixes network_expansion_options cross-referencing the non-existent constraints_rhs table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AEMO timeslice calendar (timeslice_RefYear5000.csv, shipped under templater/plexos/ with the other PLEXOS data-package extracts) marks hot day / typical summer / winter windows on absolute dates, with each planning year's dates derived from the weather of the reference year AEMO assigned to it. Templating the calendar as absolute windows would bake AEMO's reference-year sequence into every run regardless of the user's configured reference_year_cycle, so peak-day limits would fall on days that aren't peaks in the modelled traces. Instead the templater inverts the calendar: it assigns windows to the planning year they start in, labels them via Table 1 of the Draft 2026 ISP Market Model Instructions (transcribed as the manually extracted reference_year_sequence table, extended cyclically per AEMO's repeating sequence), and emits one month-day window pattern per reference year for the translator to re-sequence. Every occurrence of a reference year must carry an identical pattern — the check that proves the Table 1 decode against the data; it passes for all 33 usable planning years. Planning years truncated at the calendar horizon (windows that never turn off) are dropped whole: their reference years recur earlier with complete windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new-format templater now emits the same table set at every granularity: timeslices and costs_connection are wired in, and the custom-constraint tables are emitted header-only at nem_regions / single_region instead of being absent. This reverses the gating from 9366945 — the PLEXOS constraints are sub-regional export-group limits with no meaningful representation once sub-regions are collapsed, but writing "all columns, no rows" tables means list_templater_output_files needs no granularity awareness and downstream consumers never check for missing tables. costs_connection is also now tracked in the output list — it was being written but untracked, the same class of bug 9366945 fixed for the constraint tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
network_geography becomes buses (REZ buses dropped when rezs is attached_to_parent_node), and the path/limit/expansion tables become links. The winter_reference limit is the link's static p_nom — winter is the calendar's default season — with the other timeslices' forward and reverse limits emitted as per-unit values in a link_timeslice_limits table for pypsa_build to expand into p_max_pu/p_min_pu series (per-unit values can exceed 1.0 when a season's limit tops winter's). Asymmetric reverse limits land in p_min_pu rather than separate links. Expansion options become one extendable link per (path, investment period) with annuitised capital costs, gated by transmission_expansion / rez_transmission_expansion according to whether the path connects a REZ. Paths with no capacity data take rez_to_sub_region_transmission_default_limit, replicating the existing REZ behaviour. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The translator assigns a reference year to each model financial year with the identical construct_reference_year_mapping call the trace pipeline makes, so timeslice windows and demand/VRE traces always come from the same weather years — the binding interaction between hot-day limits and hot-day demand survives any configured cycle. The year before the first model year also gets a pattern (the cycle's last reference year, cycle-consistent) because winter windows run April-October and must cover the first model year's July-September snapshots. Month-day boundaries of 29 February (reference year 2024 carries them) clamp to 28 February in non-leap model years. The wrap decision for exclusive ends compares the month-day strings rather than the placed dates, so clamping a one-day 02-28..02-29 window collapses it to empty instead of stretching it out a year. A configured cycle requesting reference years the timeslices table has no patterns for raises rather than silently never binding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each RHS value becomes one constraint instance per investment period (and per timeslice where tagged), with date_from resolved as the value active at the start of each period. LHS terms keep generator and battery names even though those components aren't translated yet — pypsa_build skips-and-logs missing components, and the logging silences itself once generator translation lands. The generators/batteries parameters are accepted now as the future name-mapping hook. Constraint-relaxation expansion options become dummy generators on bus_for_custom_constraint_gens, with their LHS terms filtered per period by build_year so a generator built for 2040 can't relax a 2030 constraint instance — an improvement over the old single-constraint formulation, which couldn't express this. Extendable links and relaxation generators get "_expansion_limit" constraints capping total p_nom at the option's allowed expansion; the suffix keeps their names clear of the parent constraints'. Duplicate input rows and RHS rows without LHS pairs raise rather than collapse silently; the "load" term type maps to Load/p, which pypsa_build logs as unimplemented and skips, matching the old path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Links expand their per-unit timeslice limits into per-snapshot p_max_pu/p_min_pu series at network-build time, using the timeslice_snapshots mapping; snapshots outside any window keep the static (winter) value. Doing the expansion here rather than shipping per-link series from the translator keeps the pypsa-friendly directory small and reuses the same mapping table that scopes the custom constraints. New-format custom constraints build one linopy constraint per RHS row, restricting time-indexed variables to the row's investment period and timeslice via label-based selection on (period, timestamp) pairs. LHS terms for components not in the model (generators and batteries, until their translation lands) are skipped with a log line each; constraint instances with no snapshots in scope are skipped silently because the translator already warns about timeslices that never apply. build.py dispatches between the old and new constraint paths on the presence of the investment_period column. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
create_pypsa_friendly_inputs dispatches to the network-tables path under the use_new_table_format flag. Snapshots come straight from the temporal config — aggregation methods needing demand traces aren't available until generator translation lands — and the output set covers the network and custom-constraint functionality only: no generators, batteries, or timeseries yet, so the CLI skips the timeseries task and list_timeseries_files returns nothing under the flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the translator and pypsa_build up to speed with the new table formats, behind the
use_new_table_formatfeature flag. With this change the new-format pipeline runs end to end — templater → translator → PyPSA network — for the network and custom-constraint functionality. Generators, batteries and timeseries are deliberately out of scope and remain the main follow-up.Two ideas shape most of the diff. First, transmission limits and custom-constraint RHS values are timeslice-tagged in the new tables, so the model gains temporal scope: links get per-snapshot
p_max_pu/p_min_puseries built from per-timeslice limits, and each custom-constraint RHS value becomes one linopy constraint per investment period (and timeslice), rather than a single constraint over everything. Second, the timeslice calendar is decoded per reference year: AEMO's PLEXOS calendar marks hot/typical/winter windows on absolute dates following their rolling reference-year sequence, and the templater inverts that (using Table 1 of the Draft 2026 ISP Market Model Instructions) into one month-day pattern per weather year, which the translator re-sequences to the configuredreference_year_cycle— the same assignment used for demand and VRE traces, so peak-day limits land on the modelled peak days whatever cycle a user picks.Design choices a reviewer would otherwise have to reverse-engineer:
p_nomis the forward winter limit, with other timeslices as per-unit multipliers (which may exceed 1.0). Winter is the calendar's default season; snapshots outside any window keep the static value. Asymmetric reverse limits live inp_min_pu, not separate links.link_timeslice_limits+timeslice_snapshots) expanded in pypsa_build, rather than per-link series files from the translator. Same formulation, less I/O, and the mapping table doubles as the snapshot-restriction input for custom constraints.reference_year_cycleoutside the calendar's weather years (2011–2025, year-ending) raises at translation time rather than silently never binding any timeslice.The commits are chunked to be reviewable in order: schemas → templater (timeslices, wiring) → translator (network, timeslices, constraints) → pypsa_build → orchestrator/CLI. Old-format behaviour is untouched; every site to delete when the flag is retired is marked
FEATURE_FLAG_CLEANUP[use_new_table_format].Known follow-ups: generator/battery translation for the new format (which also unlocks trace-based snapshot aggregation and timeseries), demand-trace geography at coarser granularities, and operational-phase wiring.
🤖 Generated with Claude Code