Add custom-constraints-from-PLEXOS templater

[nick-gorman]

What this does

ISPyPSA needs AEMO's REZ / sub-region export limits as custom constraints, but
they don't come out of the IASR workbook in a usable shape. AEMO does express
them in its PLEXOS model, as export-group constraints, so we extract them from
there. This PR adds the templater that turns that extract into ISPyPSA's three
custom-constraint tables:

• custom_constraints: one row per constraint (id + direction)
• custom_constraints_lhs: the left-hand-side terms (which unit, what coefficient)
• custom_constraints_rhs: the per-timeslice limit on the right

It sits behind the use_new_table_format flag (committed off) and only runs at
sub_regions granularity — coarser granularities collapse the sub-region
nodes, sub-regional flow paths and REZ-located units the constraints reference,
leaving nothing to constrain.

What the translation looks like

One constraint end-to-end, based on the e2e test fixture. From the PLEXOS
extract (lhs_terms.csv, abbreviated — constraints.csv carries SWQLD1's
Sense of -1, rhs_values.csv its limit of 3000 MW tagged "QLD Hot Day"):

constraint_name     parent_class  parent_name                  property                 value
ExportGroup_SWQLD1  Generator     BW01                         Generation Sent Out C.    0.5
ExportGroup_SWQLD1  Generator     Q8_SAT_Brisbane              Generation Sent Out C.    0.4
ExportGroup_SWQLD1  Generator     SWQLD1_Linear Augmentation   Installed Capacity C.    -1.0  # relaxation term: dropped
ExportGroup_SWQLD1  Battery       SWQLD1 Battery - 2h          Generation Coefficient    1.0  # constraint-scoped: dropped, pass 2 re-supplies
ExportGroup_SWQLD1  Battery       SWQLD1 Battery - 2h          Load Coefficient         -1.0  # negative pair: dropped
ExportGroup_SWQLD1  Line          NSW1-QLD1                    Flow Coefficient          0.8
ExportGroup_SWQLD1  Purchaser     Q1 to NQ Flex. Electrolyser  Load Coefficient         -1.0  # hydrogen: dropped

the templater produces:

custom_constraints:
constraint_id  direction
SWQLD1         <=                                              # prefix stripped; Sense -1 -> <=

custom_constraints_lhs:
constraint_id  term_type         variable_name    coefficient  date_from
SWQLD1         generator_output  BW01             0.5
SWQLD1         generator_output  Q8_SAT_Brisbane  0.4
SWQLD1         link_flow         NSW-QLD          0.8          # PLEXOS line name -> path_id
SWQLD1         storage_output    Q8 Battery - 2h  1.0          # pass 2: Q8_SAT_Brisbane triggers REZ Q8
SWQLD1         storage_output    Q8 Battery - 4h  1.0          # incl. the 4h duration PLEXOS omits

custom_constraints_rhs:
constraint_id  timeslice        rhs     date_from
SWQLD1         qld_peak_demand  3000.0                         # "QLD Hot Day" -> region-prefixed timeslice

Note the RHS timeslice: it comes out region-prefixed (qld_peak_demand)
rather than bare (peak_demand) like the rest of the templater's outputs. A
follow-up PR will bring the rest of the transmission functionality onto this
format so the constraints flow through end-to-end.

The bit worth understanding first: batteries

The RHS and most of the LHS are a straight translation from the PLEXOS
extract. The one judgement call is batteries.

PLEXOS encodes battery participation as per-constraint LP variables (the
SWQLD1 Battery - 2h rows in the example above) whose layout doesn't map onto
any IASR unit (see #110), and it quietly leaves out the 4h
durations. Rather than mirror that, the templater builds the LHS in two
passes:

1. Translate. Convert the raw PLEXOS LHS rows (generators, batteries,
   lines, nodes) into ISPyPSA's term schema. Unit names that don't resolve to
   an IASR unit are dropped — which removes the constraint-scoped battery
   variants by design.
2. Re-inject. For each constraint, find the REZs / sub-regions where its
   surviving new-entrant units (generators or batteries) sit, and add a
   storage_output term for every IASR new-entrant battery at those
   locations. Each injected battery copies the (date_from, coefficient)
   profile of the new-entrant batteries that survived pass 1 there, so an
   injected 4h battery lands at the same weight as its 2h/8h siblings (e.g.
   0.43 at SWQLD1's Q8) rather than a flat 1.0. Where no sibling survived, it
   falls back to 1.0 and a WARNING names the (constraint, location) so the
   default can be audited.

The copy is only well-defined if a location's surviving batteries agree, so
the templater raises if they don't share one profile.

More generally, the templater fails loud on anything it doesn't recognise: an
unmapped (parent_class, property) pair or Sense value, an unknown PLEXOS
line name, a battery Load Coefficient row missing its Generation Coefficient pair, or a populated date_to. Deliberate drops are logged —
INFO where they're by-design (Purchaser electrolysers, constraint-relaxation
terms, battery Load pairs), WARNING where the drop loosens a constraint (units
with no ISPyPSA counterpart).

Where it lives

ISPyPSA/
├─ src/ispypsa/templater/
│  ├─ custom_constraints_from_plexos.py   the templater (PLEXOS extract → the 3 tables)
│  ├─ plexos/7.5/*.csv                    the extract it reads at runtime (already on main)
│  ├─ create_template.py                  wires it in at sub_regions; tracks the 3 outputs
│  └─ mappings.py, transmission.py        now share one canonical-timeslice vocabulary
├─ scripts/
│  ├─ extract_plexos_constraints.py       PLEXOS XML → the extract above (already on main)
│  └─ workbook_extract_constraint.py      independent workbook-route LHS, validation only
└─ tests/test_templater/
   ├─ test_custom_constraints_from_plexos.py   templater unit tests
   ├─ test_custom_constraints_validation.py    PLEXOS route vs workbook route reconciliation
   └─ data/custom_constraints_validation/      fixtures (plexos/ + workbook/)

Supporting changes

The templater is wired into create_ispypsa_inputs_template (gated to
sub_regions, its three outputs tracked as task targets); the two IASR summary
tables it needs are added to the required-tables list; and the canonical
timeslice vocabulary is promoted into mappings so this templater and the
existing transmission code share one copy.

Validation

Pulling these constraints out of PLEXOS takes enough translation that we
didn't want to take the output on trust, so the PR cross-checks the templater
against a second, fully independent reconstruction of each constraint built
straight from the IASR workbook. The two routes share no code:

• The PLEXOS route is the production templater reading the PLEXOS extract.
• The workbook route (scripts/workbook_extract_constraint.py) looks each
  constraint up in rez_group_constraint_summary, parses its Terms into
  (coefficient, body) pairs, and expands them: a REZ id (e.g. Q1) becomes
  every generator, battery and electrolyser tagged with that REZ ID across the
  IASR unit summary tables; an interconnector path id (e.g. CQ-NQ) becomes a
  single link_flow term. Coefficients can be implicit (Q1 is 1.0), signed
  (- CQ-NQ is -1.0) or explicit (0.78 * V7).

test_custom_constraints_validation.py runs both routes over five constraints
(NQ1, NET1, WV1, CQ1, MN1) and asserts the LHS term sets match
exactly, save for an explicit EXPECTED_DELTAS table. A new mismatch in
either direction fails the test, and so does an expected one going missing.

The documented differences come down to two things:

1. Geographic lookup for existing plant. Every IASR unit carries both a
   fine-grained REZ ID and a coarser sub-region, and they usually agree. The
   workbook route expands by REZ ID; PLEXOS includes existing plant by
   sub-region. The handful of boundary plants where the two tags point
   different places are the only generator diffs — e.g. EMERASF1 (REZ Q4,
   sub-region NQ) lands in NQ1 under PLEXOS but CQ1 under the workbook,
   so the one fact shows up on both sides of the comparison.
2. Electrolysers. PLEXOS routes electrolyser load through its Purchaser
   class, which the templater drops by design (hydrogen demand isn't
   modelled), so they appear as workbook-only load terms.

Batteries don't surface as a difference: the five validated constraints are
all unit-coefficient, so both routes land the same batteries at 1.0. The
pass-2 coefficient copy only bites on the fractional constraints (SWQLD1,
WNV1, NSA1, SQ1), which the workbook route can't reach — those rely on the
per-module unit tests instead.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/ispypsa/iasr_table_caching/local_cache.py	74.28% <ø> (ø)
src/ispypsa/templater/create_template.py	92.77% <100.00%> (+3.16%)	⬆️
...spypsa/templater/custom_constraints_from_plexos.py	100.00% <100.00%> (ø)
src/ispypsa/templater/mappings.py	100.00% <100.00%> (ø)
src/ispypsa/templater/transmission.py	98.70% <100.00%> (-0.01%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[EllieKallmier]

I think this looks good! My main comments/thoughts on this PR are mostly around my own confusion with some of the descriptions of stuff and phrasing used - hopefully my comments below clarify but ofc happy to discuss more.

The two-pass approach makes sense, and matches up with behaviour elsewhere re: essentially merging in the 'missing' rows where there's some indication that they should exist (and no indication that they explicitly shouldn't).

I don't think any of my comments should stop this PR from merging - I'm hoping I've understood at least the main vibes of what's going on and based on that understanding this work seems solid :)

[EllieKallmier]

To clarify/summarise - this is a helper script whose output is just to print the LHS unit rows processed for the given constraint ID?

[EllieKallmier]

This I guess supports an approach to regional granularity (for the moment) where something stops users from actually running any granularity other than subregional (e.g. feature flags).

Also was just having a look back at this issue re: naming conventions and I think "subregions" (no hyphen/underscore) was the decision there for node naming - absolutely the most minor comment and I have not been paying attention to this much myself but just chucking here in case we want to chat/change/do anything in relation to how we refer to sub(-)regions in other contexts?

[EllieKallmier]

Here and for batteries below - why no DN2? (Guessing this DREZ isn't present in the plexos constraints?)

[EllieKallmier]

I think this file is where that thing you mentioned in the meeting last week about phrasing getting a little confusing/maybe odd is coming through for me. Some of the words ('trigger', 'surviving') do make sense and aren't not understandable, but maybe could be introduced or defined a bit more specifically before going on to be used kind of ubiquitously?

Just feeling a bit like sometimes adding a few extra words or something could make some of the docstrings a little easier to intuit - but of course this is a personal take and appreciate is a bit in the weeds!

To give an example, this sentence from one of the docstrings:

Surviving batteries trigger as well as generators so a location
whose generators were all dropped (or that PLEXOS includes via batteries
alone) still gets its omitted durations re-added.

is kind of isolated from previous introduction of the 'surviving' and 'trigger' terms; adding a few extra words to contextualise is prob my instinct from a clarity POV, e.g. "Surviving batteries trigger the creation of additional rows even in locations whose generators were all dropped ...".

This is not a big deal and I was able to wrap my head around this file in the end but yeah I think the use of some of these terms or phrases that have a kind of implied context-specific meaning just added some extra time for me in unpicking it. Again - prob a person-by-person thing so no change required from me, just feeding back on what we chatted about the other day!

[EllieKallmier]

No real comment just some out-loud thinking for me to reflect on:

Makes sense to drop and log as warning here, and/but kinda relates to the canonicalisation question you raised in #107 and our discussions around what we expect the templater input to look like. Basically I'm wondering if (as you posited in a comment on that pr) it might make more sense or be simpler to just drop/warn/not raise on non-matches and let the validator raise 'real' problems later (i.e. not just 'not implemented yet' kinda things)... in some cases

[EllieKallmier]

I am getting a little bit mixed up with some of the wording here and what the _raise_on_inconsistent_battery_profiles function below actually checks for 'inconsistency' - very possible I'm missing something about constraint definitions so grain of salt!

Specifically:

• "the surviving new-entrant batteries must share one (date_from, coefficient) profile" -> reads to me like saying only one (date_from, coefficient) tuple is allowed for each (constraint, location) tuple - in which case I think I assume that the I/O example should raise for the (WNV1, V7) pair?
• "a time-varying sibling whose extra date_from rows would otherwise be grafted onto its constant siblings silently" (below) -> describes a failure model -> to me reads like it's saying only either time varying or constant (but not both) pairs allowed, which again the I/O example here would raise (?)

From the tests and the code itself I can see this is not the correct interpretation - think maybe these two docstrings and/or example here could be edited a little to improve clarity?

[EllieKallmier]

I'm a bit confused by the phrasing "0.78 always, 0.0 from date" - possibly just not understanding constraint formulation but "always" then "from x" seem contradictory? Or jsut not very clear - like "0.78 up to date, 0.0 from date" would be more obvious to me (related to earlier comment/confusion around this function's definition) - but perhaps I've also got an incorrect interpretation of what these values mean

[EllieKallmier]

Ok I think this explains this function a bunch more for me - I understand the vibe here!

Still though think there could be some clarification within the docstrings/examples to make this a bit more intuitive or obvious there?