Custom-constraints templater: state of play and known PLEXOS-vs-IASR gaps

[nick-gorman]

TL;DR

This issue is the context document the templater source points at via
Open-ISP/ISPyPSA#111. It captures:

• The PLEXOS-vs-IASR data-model gaps that drive
  custom_constraints_from_plexos.py's design choices (boundary plants
  • constraint-scoped battery layout; the underlying battery inventory data is
    in Open-ISP/ISPyPSA#110).
• The two-pass build (PLEXOS translation + IASR new-entrant battery
  injection) we settled on, and the one prefix-rename (DN1/DN3 →
  DREZ) we kept.
• An annotated walk-through of the INFO/WARNING log lines the templater
  emits on a full IASR cache run.

Not a bug report — a state-of-play record. The "Open questions" section
at the end lists templater-specific items that would benefit from a
clarification.

Background: PLEXOS and IASR use two different location systems

Each IASR generator/battery carries two geographic tags:

• REZ ID (Q1, Q2, …, S3, S4, …) — fine-grained, ~50 zones.
• Sub-region (NQ, CQ, NSA, CSA, …) — coarser, ~22 areas.

Usually a REZ sits cleanly inside one sub-region. PLEXOS' export
constraints (ExportGroup_*) mix the two systems — their generator
terms are mostly tagged at REZ level (Q8, S5, T1, …) but their
battery terms historically used per-constraint sub-region-style aliases
(SWQLD1 Battery - 2h, NSA1 Battery - 8h, …). The templater has to
reconcile both vocabularies into ISPyPSA's IASR-keyed unit space.

Terminology: "constraint-scoped variant"

The rest of this issue uses the term constraint-scoped variant for a
recurring pattern in PLEXOS' export constraints. A constraint-scoped
variant is a PLEXOS Battery object that:

• has a constraint-scoped name — its prefix matches exactly one
  ExportGroup_* constraint (e.g. SWQLD1 Battery - 2h belongs to
  ExportGroup_SWQLD1);
• is a member of that one constraint's LHS with
  Generation Coefficient = 1.0;
• co-exists with one or more sibling Battery objects at the same
  Node with different names and otherwise-identical physical properties
  (Capacity, Max Power, efficiencies, FO&M, etc.) — the sibling is
  not in the export constraint.

Its structural role in the LP is to give the export constraint a
battery LHS term; its sibling carries (or doesn't) the actual buildable
storage capacity. The term describes that structural role, not the
variant's buildability:

• For some groups (SWQLD1 at 8h, MN1 at 8h, …) the constraint-scoped
  variant's Max Units Built is nonzero and the constraint binds on
  real capacity attached to that LP variable.
• For others (every group at 2h; NET1/SWV1 at any duration) the
  constraint-scoped variant's Max Units Built is zero, so the
  constraint binds on an LP variable that can never carry capacity.

Which configuration applies in each case is documented in
#110.

Throughout the rest of this issue:

• "constraint-scoped variant" → the PLEXOS Battery object whose name is
  scoped to a single export constraint.
• "sub-region-named variant" / "sub-region-named battery" → the
  sibling that shares the constraint-scoped variant's Node but uses the
  IASR sub-region prefix (e.g. SQ Battery - 2h next to
  SWQLD1 Battery - 2h).
• "the seven constraint-scoped groups" / "the constraint-scoped set" →
  the collection across the seven prefixes: SWQLD1, SQ1, MN1,
  NSA1, NET1, SEVIC1, SWV1.

Gap A: REZ-vs-sub-region boundary plants

The handful of existing/committed plants whose REZ and Sub-region tags
don't line up cleanly. PLEXOS includes them by Sub-region; the workbook
expands by REZ; the two recipes disagree on a few units:

Plant	REZ ID	Sub-region	Workbook places in	PLEXOS places in
EMERASF1 (Emerald SF)	Q4	NQ	CQ1	NQ1
CSPVPS1 (Collinsville SF)	Q10	CQ	—	CQ1
DAYDSF1 (Daydream SF)	Q10	CQ	—	CQ1
WPWF (Wattle Point WF)	S4	NSA	MN1	—

These are the residual diffs in
tests/test_templater/test_custom_constraints_validation.py::EXPECTED_DELTAS
after the two-pass templater runs. ISPyPSA currently inherits PLEXOS'
choice (Sub-region-based inclusion for existing plants), since that's
the source custom_constraints_from_plexos.py reads from.

Gap B: PLEXOS' export-constraint batteries are mostly constraint-scoped variants

PLEXOS uses the parallel-LP-variable pattern (see "Terminology" above)
extensively in its battery modelling. The templater-relevant slice is
the seven constraint-scoped groups that show up as battery LHS
terms in the ExportGroup_* constraints:

PLEXOS constraint-scoped prefix	Maps to sub-region IASR battery
SWQLD1	SQ
SQ1	SQ
MN1	CSA
NSA1	NSA
NET1	TAS
SEVIC1	SEV
SWV1	WNV

What matters for the templater design:

• These constraint-scoped variants don't match any IASR battery
  name, so they can't be resolved through a literal rename. A naïve
  constraint-scoped → sub-region-named rename (which we tried first)
  misrepresents PLEXOS' LP structure — the sub-region-named LP variable
  PLEXOS actually carries is a separate LP variable that isn't in the
  constraint at all, and Max Units Built on each side varies per group
  and per duration in ways that don't follow one simple rule.
• The constraint-scoped set only covers 2h and 8h durations. There
  are no SWQLD1 Battery - 4h (or 1h, or Distributed Resources)
  objects, so the export constraints can't bind on those durations at
  all even though IASR ships batteries at every duration.
• The 2h side of the constraint-scoped set is uniformly
  Max Units Built = 0 (so the 2h LHS term of every export constraint
  is on an LP variable that can never carry capacity). The 8h side
  splits four ways across the seven groups, from "the constraint-scoped
  variant carries 50000 of buildable capacity" to "nothing buildable at
  either duration".

The full detail — including the cross-PLEXOS / cross-IASR comparison
tables, the four-way 8h split, the same zero-build LP-variable pattern
showing up on REZ-named batteries too, and what we ruled out as
non-obvious linking mechanisms — lives in #110.

The templater's job here is to do something sensible for ISPyPSA
without trying to mirror PLEXOS' constraint-scoped layout literally.
The "How ISPyPSA handles it" section below is the design we landed on.

How ISPyPSA handles it now

src/ispypsa/templater/custom_constraints_from_plexos.py builds the LHS
in two passes:

Pass 1 — PLEXOS → ISPyPSA translation

Standard term-by-term translation:

• Generator + Generation Sent Out Coefficient → generator_output.
• Battery + Generation Coefficient → storage_output.
• Line + Flow Coefficient → link_flow (via the hardcoded
  _LINE_TO_PATH_ID table).
• Node + Load Coefficient → load.

Names are matched case-insensitively against the IASR unit summary
tables. The constraint-scoped battery prefixes from Gap B
(SWQLD1, SQ1, MN1, NSA1, NET1, SEVIC1, SWV1) have no
rename rule — they don't match any IASR battery name and fall through
_drop_unresolved_terms. Pass 2 supplies the replacement.

The DREZ correction (retained from pass 1)

DN1 / DN3 → DREZ is the only battery prefix rename retained.
Unlike the seven Gap-B constraint-scoped variants, this is a one-to-one
naming-style mapping:
DN1/DN3 are the literal IASR REZ IDs, and DREZ is IASR's naming
prefix for batteries inside REZs DN1/DN2/DN3. Keeping the rename in
pass 1 lets us preserve PLEXOS' time-varying coefficients (e.g. the
0.20–0.35 schedule on the CNSW-SNW South GPG constraint) — pass 2's
flat 1.0 injection wouldn't reproduce that.

Pass 2 — IASR new-entrant battery injection

The injection is what closes the gaps PLEXOS' constraint-scoped set
leaves: it supplies the IASR new-entrant batteries the export
constraints should bind on but PLEXOS' constraint-scoped variants
either misrepresent, omit entirely (no 4h / 1h / Distributed Resources
constraint-scoped objects in any export constraint), or render
unbuildable through the per-group Max Units Built choices. See
#110 for the underlying coverage data.

For each constraint, _inject_iasr_new_entrant_batteries does:

1. Collect the surviving generator_output terms.
2. For each, look up the unit in new_entrants_summary and resolve its
   location — REZ ID when populated, else Sub-region.
3. For each triggered location, add one storage_output row per IASR
   new-entrant battery at that same location — covering every
   duration IASR ships for that location (REZ-level: 2h/4h/8h;
   sub-region-level: 1h/2h/4h/8h + Distributed Resources). Each gets
   coefficient = 1.0 (matches PLEXOS' Generation Coefficient on its
   in-constraint variants) and empty date_from. The constraint
   therefore binds on storage MW with no duration weighting.

Then _dedupe_lhs_terms collapses any pass-1/pass-2 collisions on the
full (constraint_id, term_type, variable_name, date_from) key,
keeping pass-1 first (so PLEXOS' time-varying coefficients win).

Net behavioural change vs the previous all-prefixes-rename approach:
+33 LHS rows added (overwhelmingly the missing-4h REZ batteries —
N3, N9, Q1–Q4, Q7, Q8, Q10, S3–S8, T1, V3, V4, V5, V7 at 4h, plus a
few REZ batteries PLEXOS missed entirely), −14 removed (the
seven constraint-scoped mappings, dropped), 0 coefficient changes.

What you'll see in the logs

A clean run of template_custom_constraints_from_plexos against the
full IASR cache emits the following INFO/WARNING lines. Each is
annotated below.

INFO Creating a custom-constraints template from the PLEXOS extract

Start banner. Always fires.

---

INFO Dropped 30 LHS rows for excluded parent classes ['Purchaser']: parents=['Q1 in NQ Baseload for Electrolyser', 'Q1 to GG Flexible Electrolyser Purchaser Green Commodities', 'Q1 to NQ Flexible Electrolyser Purchaser Domestic', 'Q4 in CQ Baseload for Electrolyser', 'Q4 to CQ Flexible Electrolyser Purchaser Domestic', 'Q4 to GG Flexible Electrolyser Purchaser Green Commodities', 'Q7 in SQ Baseload for Electrolyser', 'Q7 to GG Flexible Electrolyser Purchaser Green Commodities', 'Q7 to SQ Flexible Electrolyser Purchaser Domestic', 'S3 in CSA Baseload for Electrolyser', 'S3 to CSA Flexible Electrolyser Purchaser Domestic', 'S3 to NSA Flexible Electrolyser Purchaser Green Commodities', 'S7 in NSA Baseload for Electrolyser', 'S7 to NSA Flexible Electrolyser Purchaser Domestic', 'S7 to NSA Flexible Electrolyser Purchaser Green Commodities', 'T1 in TAS Baseload for Electrolyser', 'T1 to TAS Flexible Electrolyser Purchaser Domestic', 'T1 to TAS Flexible Electrolyser Purchaser Green Commodities', 'V5 in WNV Baseload for Electrolyser', 'V5 to WNV Flexible Electrolyser Purchaser Domestic', 'V5 to WNV Flexible Electrolyser Purchaser Green Commodities', 'V7 in WNV Baseload for Electrolyser', 'V7 to WNV Flexible Electrolyser Purchaser Domestic', 'V7 to WNV Flexible Electrolyser Purchaser Green Commodities']

Hydrogen electrolysers routed via PLEXOS' Purchaser class.
Intentional drop — ISPyPSA doesn't currently model hydrogen demand
(#104). No operator action.

---

INFO Dropped 42 constraint relaxation LHS rows (PLEXOS Installed Capacity Coefficient): ['CNSW1_DN1 Option 1', 'CNSW1_DN1 Option 2a', 'CNSW1_DN1 Option 2b', 'CNSW1_DN1 Option 3', 'CQ1_CQ-NQ Option 3 Augmentation', 'CQ1_CQ-NQ Option 4 Augmentation', 'CQ1_Linear Augmentation 1', 'CQ1_Linear Augmentation 2', 'MN1_CSA-NSA Option 1 Augmentation', 'MN1_CSA-NSA Option 2 Augmentation', 'MN1_CSA-NSA Option 3 Augmentation', 'MN1_CSA-NSA Option 4 Stage 1 Augmentation', 'MN1_CSA-NSA Option 4 Stage 2 Augmentation', 'MN1_Linear Augmentation 1', 'MN1_Linear Augmentation 2', 'MN1_MN1 Option 1', 'NET1_Linear Augmentation 1', 'NET1_Linear Augmentation 2', 'NQ1_CQ-NQ Option 1 Augmentation', 'NQ1_CQ-NQ Option 2 Augmentation', 'NQ1_Linear Augmentation 1', 'NQ1_Linear Augmentation 2', 'NSA1_Linear Augmentation 1', 'SEVIC1_SEVIC1 Option 1', 'SEVIC1_V8 Option 2', 'SQ1_Linear Augmentation 1', 'SQ1_Linear Augmentation 2', 'SQ1_SQ-CQ Option 5 Augmentation', 'SQ1_SQ-CQ Option 6 Augmentation', 'SWNSW1_Project EnergyConnect Augmentation', 'SWNSW1_SWNSW1 Option 1', 'SWNSW2_SNSW-CNSW Option 3 Augmentation', 'SWNSW2_SNSW-CNSW Option 4 Augmentation', 'SWQLD1_Linear Augmentation 1', 'SWQLD1_Linear Augmentation 2', 'SWV1_Linear Augmentation 1', 'SWV1_Linear Augmentation 2', 'SWV1_SWV1 Option 1A', 'WV1_Linear Augmentation 1', 'WV1_WNV-SNSW Option 1 - VNI West (excluding N5 option 1) Augmentation', 'WV1_WNV-SNSW Option 1 - VNI West Augmentation', 'WV1_Western Renewables Link']

PLEXOS attaches "Linear Augmentation" and named "Option …" generator
rows to each export constraint via the Installed Capacity Coefficient
property. These are the PLEXOS solver's constraint-relaxation /
expansion-option mechanism, not operational LP terms. ISPyPSA's network
expansion is handled in a separate templater
(network_expansion.py), so these get dropped here. No operator action.

---

INFO Applied 26 PLEXOS->ISPyPSA LHS name renames: ['CNSW Battery - Distributed Resources Area1 -> CNSW Battery - Distributed Resources', 'CNSW SAT - Distributed Resources Area1 -> CNSW SAT - Distributed Resources', 'DN1 Dubbo Battery - 2h -> DREZ Dubbo Battery - 2h', 'DN1 Dubbo Battery - 4h -> DREZ Dubbo Battery - 4h', 'DN1 Dubbo Battery - 8h -> DREZ Dubbo Battery - 8h', 'DN1_SAT_Dubbo -> DREZ_SAT_Dubbo', 'DN1_WH_Dubbo -> DREZ_WH_Dubbo', 'DN3 Marulan Battery - 2h -> DREZ Marulan Battery - 2h', 'DN3 Marulan Battery - 4h -> DREZ Marulan Battery - 4h', 'DN3 Marulan Battery - 8h -> DREZ Marulan Battery - 8h', 'DN3_SAT_Marulan -> DREZ_SAT_Marulan', 'DN3_WH_Marulan -> DREZ_WH_Marulan', 'EnergyConnect -> SNSW-CSA', 'Marinus -> TAS-SEV', 'NSW1-QLD1 -> NSW-QLD', 'Q2_CST_North QLD Clean Energy Hub -> Q2_CST_North Qld Clean Energy Hub', 'Q2_SAT_North QLD Clean Energy Hub -> Q2_SAT_North Qld Clean Energy Hub', 'Q2_WH_North QLD Clean Energy Hub -> Q2_WH_North Qld Clean Energy Hub', 'Q2_WM_North QLD Clean Energy Hub -> Q2_WM_North Qld Clean Energy Hub', 'Q3_CST_Northern QLD -> Q3_CST_Northern Qld', 'Q3_SAT_Northern QLD -> Q3_SAT_Northern Qld', 'Q3_WH_Northern QLD -> Q3_WH_Northern Qld', 'Q3_WM_Northern QLD -> Q3_WM_Northern Qld', 'T-V-MNSP1 -> TAS-SEV', 'V-SA -> WNV-SESA', 'VIC1-NSW1 -> WNV-SNSW']

Every systematic PLEXOS→IASR name rename applied in pass 1. Four
patterns visible here:

• <name> Area1 -> <name> — Area-suffix strip (only Area1 appears
  in the current PLEXOS extract).
• DN1/DN3 ... -> DREZ ... — the DREZ correction (see "How ISPyPSA
  handles it" above), applied to both generator and battery names.
• Q2/Q3 ... QLD ... -> ... Qld ... — case mismatch fix.
• Line names — PLEXOS line ID → ISPyPSA path_id (NSW1-QLD1 -> NSW-QLD,
  Marinus -> TAS-SEV, etc.).

No operator action. Useful for audit when a future PLEXOS rev adds a
new naming pattern we haven't accounted for.

---

WARNING Dropped 31 LHS term rows (19 distinct units) with no ISPyPSA unit match: ['CNSW-SNW South GPG: CNSW Battery - Coordinated CER Storages Area1', 'CNSW-SNW South GPG: CNSW V2G Area1', 'CNSW-SNW South GPG: PV CNSW Area1', 'MN1: MN1 Battery - 2h', 'MN1: MN1 Battery - 8h', 'NET1: NET1 Battery - 2h', 'NET1: NET1 Battery - 8h', 'NSA1: Cultana Solar Farm', 'NSA1: NSA1 Battery - 2h', 'NSA1: NSA1 Battery - 8h', 'NSA1: SA Hydrogen Turbine', 'SEVIC1: SEVIC1 Battery - 2h', 'SEVIC1: SEVIC1 Battery - 8h', 'SQ1: SQ1 Battery - 2h', 'SQ1: SQ1 Battery - 8h', 'SWQLD1: SWQLD1 Battery - 2h', 'SWQLD1: SWQLD1 Battery - 8h', 'SWV1: SWV1 Battery - 2h', 'SWV1: SWV1 Battery - 8h']

The meaningful drop list. The 19 distinct units fall into three
categories:

• Constraint-scoped variants (14 of 19):
  <constraint-scoped-prefix> Battery - 2h/8h for the seven groups in
  Gap B (MN1, NET1, NSA1, SEVIC1, SQ1, SWQLD1, SWV1).
  Intentional — pass 2's IASR new-entrant injection supplies the
  replacement. No operator action.
• CER / V2G / Coordinated CER Storages (3 of 19): Distributed
  consumer-energy-resources units. These exist in IASR's
  consumer_energy_resources_summary table, which ISPyPSA doesn't
  currently template (tracked in CER units participate in custom constraints — revisit name-matching when CER new entrants are added #104) — so the
  templater can't resolve them and drops them. Loosens the constraint
  relative to PLEXOS. No immediate action; revisit when CER is modelled.
• Specific named plants absent from IASR (2 of 19): Cultana Solar Farm and SA Hydrogen Turbine on NSA1. Could be a real IASR data
  gap (plant exists in PLEXOS but missing from
  existing_committed_anticipated_additional_generator_summary), a
  workbook-extraction omission, or these are non-summary units we
  haven't surfaced yet. Worth a follow-up to confirm which.

Because this is a WARNING (each drop loosens the corresponding
constraint), any change in this list should get a human eyeball — the
shape is stable enough that diffs against this baseline are meaningful.

---

INFO Injected 80 new-entrant battery storage_output rows across 13 constraints (74 distinct batteries)

Pass-2 summary. The 13 constraints excludes the two constraints whose
LHS contains no triggering new-entrant generator (SWNSW1 and
CNSW-SNW South GPG's non-Dubbo portions) — those don't fire the
injection. No operator action; useful for tracking when the upstream
PLEXOS extract changes the set of constraints.

---

INFO Deduped 49 overlapping LHS rows (pass-1/pass-2 collisions)

The dedup step. Most of these are pass-2 injections that collide with a
pass-1 row at the same (constraint_id, term_type, variable_name, date_from) — typically a REZ battery that PLEXOS already included via
some other path, or the DREZ batteries pass-1 catches and pass-2 also
attempts to inject. Pass-1 rows always win (and carry the PLEXOS
coefficient). No operator action.

Open questions

Questions about PLEXOS' battery layout itself (intentional vs
oversight, Not set Max Units Built semantics, etc.) live in
#110 rather than here, since they're properties
of the PLEXOS file independent of how the templater uses it. The
questions below are the ones that affect the templater's behaviour
specifically.

1. Are Cultana Solar Farm and SA Hydrogen Turbine expected
   omissions from
   existing_committed_anticipated_additional_generator_summary?
   They appear in PLEXOS' NSA1 LHS but don't resolve to any IASR
   unit, so the templater drops them with the WARNING above.

2. Is PLEXOS' inclusion of EMERASF1 in NQ1 (vs the workbook's
   CQ1) the right boundary call? EMERASF1 sits in REZ Q4 (CQ
   sub-region's REZ) but its operational sub-region is NQ. Same
   question applies to CSPVPS1/DAYDSF1 (REZ Q10, Sub-region CQ —
   PLEXOS includes in CQ1, workbook can't) and WPWF (REZ S4,
   Sub-region NSA — workbook includes in MN1, PLEXOS doesn't). We
   inherit PLEXOS' choice; confirming that's the intended behaviour
   would let us close EXPECTED_DELTAS's remaining workbook-vs-PLEXOS
   diffs.