Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
371d90d
feat: initial external model support
CypherNaught-0x Feb 4, 2026
207ba07
feat: support reference images for external models
CypherNaught-0x Feb 17, 2026
7671304
fix: sorting lint error
CypherNaught-0x Feb 17, 2026
3c83692
feat: add Seedream provider and capability-driven settings visibility
Pfannkuchensack Feb 21, 2026
19650f6
feat: initial external model support
CypherNaught-0x Feb 4, 2026
74ecc46
feat: support reference images for external models
CypherNaught-0x Feb 17, 2026
a9d3b4e
fix: sorting lint error
CypherNaught-0x Feb 17, 2026
1b43769
chore: hide Reidentify button for external models
CypherNaught-0x Feb 27, 2026
5d34eab
review: enable auto-install/remove fro external models
CypherNaught-0x Feb 27, 2026
6fe1a6f
feat: show external mode name during install
CypherNaught-0x Feb 27, 2026
d5a6283
review: model descriptions
CypherNaught-0x Feb 27, 2026
0dd7283
review: implemented review comments
CypherNaught-0x Feb 27, 2026
dc665e0
review: added optional seed control for external models
CypherNaught-0x Feb 27, 2026
c6b0d45
chore: fix linter warning
CypherNaught-0x Feb 27, 2026
b91a156
review: save api keys to a seperate file
CypherNaught-0x Feb 27, 2026
3620541
docs: updated external model docs
CypherNaught-0x Feb 27, 2026
10729f4
chore: fix linter errors
CypherNaught-0x Feb 27, 2026
689725c
Merge branch 'main' into external-models
Pfannkuchensack Mar 7, 2026
f39456e
Merge branch 'main' into external-models
Pfannkuchensack Mar 12, 2026
519575e
fix: sync configured external starter models on startup
CypherNaught-0x Mar 17, 2026
757bd3d
feat(ui): add provider-specific external generation nodes
CypherNaught-0x Mar 17, 2026
bafce41
feat: expose external panel schemas in model configs
CypherNaught-0x Mar 17, 2026
257994f
feat(ui): drive external panels from panel schema
CypherNaught-0x Mar 17, 2026
c3a482e
docs: sync app config docstring order
CypherNaught-0x Mar 17, 2026
40f02aa
feat: add gemini 3.1 flash image preview starter model
CypherNaught-0x Mar 17, 2026
20a400c
feat: update gemini image model limits
CypherNaught-0x Mar 17, 2026
9e4d0bb
fix: resolve TypeScript errors and move external provider config to a…
Pfannkuchensack Mar 18, 2026
8375f95
feat: add resolution presets and imageConfig support for Gemini 3 models
Pfannkuchensack Mar 19, 2026
d8d0ebc
Remove unused external model fields and add provider-specific parameters
Pfannkuchensack Mar 20, 2026
b83b5e8
feat: initial external model support
CypherNaught-0x Feb 4, 2026
728c627
feat: add Seedream provider and capability-driven settings visibility
Pfannkuchensack Feb 21, 2026
7a53dd0
Add Seedream provider, starter models, and provider-specific UI options
Pfannkuchensack Mar 20, 2026
c35e0ae
Merge branch 'feat/external-model-settings-visibility' of https://git…
Pfannkuchensack Mar 20, 2026
27fc650
Merge branch 'main' into external-models
Pfannkuchensack Mar 23, 2026
edde0b4
Merge branch 'main' into external-models
Pfannkuchensack Mar 28, 2026
18315db
Chore Ruff check & format
Pfannkuchensack Mar 28, 2026
813a5e2
Chore typegen
Pfannkuchensack Mar 28, 2026
c2016bc
feat: full canvas workflow integration for external models
Pfannkuchensack Apr 6, 2026
4cbd60b
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 6, 2026
089e2db
Chore typegen Linux seperator
Pfannkuchensack Apr 6, 2026
3e9e052
feat: full canvas workflow integration for external models
Pfannkuchensack Apr 6, 2026
853c3ef
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 7, 2026
17157d7
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 12, 2026
ec90b2f
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 12, 2026
5c09c82
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 13, 2026
8f00759
Chore pnpm fix
Pfannkuchensack Apr 13, 2026
ec4b87b
add missing parameter
Pfannkuchensack Apr 13, 2026
cd88865
Merge branch 'main' into external-models
Pfannkuchensack Apr 14, 2026
c743106
Merge remote-tracking branch 'upstream/main' into external-models
Pfannkuchensack Apr 14, 2026
450ba7b
Merge branch 'main' into external-models
Pfannkuchensack Apr 14, 2026
98ce0b5
Merge branch 'external-models' into feat/seedream-provider
Pfannkuchensack Apr 14, 2026
b739af1
docs: add External Models guide with Gemini and OpenAI provider pages
Pfannkuchensack Apr 14, 2026
78a1863
Merge branch 'external-models' into feat/seedream-provider
Pfannkuchensack Apr 14, 2026
a16452f
docs: add Seedream provider page to External Models guide
Pfannkuchensack Apr 14, 2026
d3ec413
chore: ruff format seedream provider and starter_models
Pfannkuchensack Apr 14, 2026
a22efb8
chore(frontend): regenerate schema.ts to include Seedream config fields
Pfannkuchensack Apr 14, 2026
f2c6c82
fix: remove reserved Windows device name "nul" from repo
Pfannkuchensack Apr 14, 2026
0b94fab
Chore windows path
Pfannkuchensack Apr 14, 2026
9bd4008
fix: update InvokeAIAppConfig docstring with Seedream config fields
Pfannkuchensack Apr 14, 2026
bf71cf1
chore(frontend): regenerate schema.ts after docstring update
Pfannkuchensack Apr 14, 2026
73b15a2
fix(external-models): address PR review feedback
Pfannkuchensack Apr 15, 2026
dd34155
Merge branch 'external-models' into feat/seedream-provider
Pfannkuchensack Apr 15, 2026
e9cd76d
fix(seedream): TSC errors + seedream provider follow-ups
Pfannkuchensack Apr 15, 2026
d751492
chore pnpm fix
Pfannkuchensack Apr 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 199 additions & 0 deletions 3-LAYER-FLATTENING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
# 3-Layer Flattening System — Design Document

## Problem

Each canvas entity (raster layer, control layer, inpaint mask, regional guidance) creates its own `Konva.Layer`, which in turn creates a separate HTML `<canvas>` element in the DOM. With many layers, the browser must composite all of these canvas elements on every frame, leading to significant GPU/CPU overhead and sluggish interactions.

## Goal

Reduce the number of active `<canvas>` elements from **N** (one per entity) to **3** (constant), regardless of how many entities exist. This provides a dramatic performance improvement especially on lower-end devices.

## Architecture

### The 3-Layer System

```
┌─────────────────────────────────────┐
│ Konva.Layer: "behind" │ ← All entities below the active one,
│ (flattened composite of layers │ flattened into a single canvas
│ behind the active entity) │
├─────────────────────────────────────┤
│ Konva.Layer: "active" │ ← The currently selected entity,
│ (the entity being edited) │ fully interactive with all
│ │ sub-modules (transformer, etc.)
├─────────────────────────────────────┤
│ Konva.Layer: "ahead" │ ← All entities above the active one,
│ (flattened composite of layers │ flattened into a single canvas
│ above the active entity) │
└─────────────────────────────────────┘
```

Plus the existing **background layer** and **preview layer** (bbox, staging area, tool) which are unchanged.

### Key Concepts

**Flattening**: Render multiple entity layers into a single off-screen canvas, then display that canvas as a single `Konva.Image` node on the composite `Konva.Layer`. This is similar to what `CanvasCompositorModule.getCompositeCanvas()` already does for generation.

**Active Entity**: The entity currently selected by the user. This entity keeps its own dedicated `Konva.Layer` so it can be interactively edited (brush strokes, transforms, filters, SAM segmentation, etc.).

**Re-flattening**: When the user switches the active entity, the "behind" and "ahead" composites must be regenerated. This can be done incrementally (add/remove one entity from composite) or fully (re-render all).

## Implementation Plan

### Phase 1: New Module — `CanvasLayerFlatteningModule`

Create a new module at `konva/CanvasLayerFlatteningModule.ts`:

```typescript
class CanvasLayerFlatteningModule extends CanvasModuleBase {
// The two composite Konva.Layers
behindLayer: Konva.Layer;
aheadLayer: Konva.Layer;

// Cached composite canvases
behindCanvas: HTMLCanvasElement | null;
aheadCanvas: HTMLCanvasElement | null;

// Konva.Image nodes to display the composites
behindImage: Konva.Image;
aheadImage: Konva.Image;

// Track which entity is active
activeEntityId: string | null;
}
```

**Responsibilities:**

1. Subscribe to `selectedEntityIdentifier` changes
2. On entity selection change, re-flatten the "behind" and "ahead" composites
3. Manage the two composite `Konva.Layer` nodes on the stage
4. Ensure individual entity adapters DON'T add their own `Konva.Layer` to the stage (except the active one)

### Phase 2: Modify Entity Adapter Lifecycle

**Current flow** (in `CanvasEntityAdapterBase` constructor):

```typescript
this.konva = {
layer: new Konva.Layer({ ... }),
};
this.manager.stage.addLayer(this.konva.layer);
```

**New flow:**

- Entity adapters still create a `Konva.Layer` (needed for rendering to off-screen canvas via `getCanvas()`)
- But they do NOT add it to the stage by default
- Only the **active entity** has its layer added to the stage
- The flattening module manages which entity is "live" on stage

### Phase 3: Composite Rendering

Reuse the existing compositing logic from `CanvasCompositorModule.getCompositeCanvas()`:

```typescript
flattenBehind(activeIndex: number): HTMLCanvasElement {
const behindAdapters = this.getOrderedAdapters().slice(0, activeIndex);
// Filter to only enabled/visible adapters
const canvas = document.createElement('canvas');
// ... render each adapter's getCanvas() onto the composite
return canvas;
}
```

**Blend modes**: Each raster layer can have a `globalCompositeOperation`. When flattening, these must be applied in order during compositing (same as `getCompositeCanvas` already does).

**Opacity**: Each layer's opacity must be respected during compositing.

**Adjustments**: Per-layer adjustments (brightness, contrast, curves) must be baked into the flattened result.

### Phase 4: Incremental Updates

When only the active entity changes content (brush strokes, image generation), the "behind" and "ahead" composites don't need to change. This is the common case and should be fast.

When a non-active entity changes (rare during editing), the affected composite must be regenerated. This can be detected via entity state subscriptions.

**Cache invalidation strategy:**

- Hash the state of all entities in each composite (similar to `getCompositeHash()` in compositor)
- Only re-flatten when the hash changes
- Cache the flattened canvas in the `CanvasCacheModule`

### Phase 5: Entity Selection Switch

When the user selects a different entity:

1. Remove the previously active entity's `Konva.Layer` from the stage
2. Render the previously active entity into the appropriate composite (behind or ahead)
3. Extract the newly active entity from its composite
4. Add the newly active entity's `Konva.Layer` to the stage
5. Re-render both composites without the newly active entity
6. Restore z-order: behind → active → ahead

**Optimization**: If the new selection is adjacent to the old one, only one composite needs to change by adding/removing one entity.

### Phase 6: Handle Edge Cases

**Entity types across composites:**
The draw order is: raster layers → control layers → regions → inpaint masks. All entity types participate in flattening. The "behind" composite includes all entities below the active one regardless of type, and "ahead" includes all above.

**Isolated preview modes:**
When filtering/transforming/segmenting, only the active entity should be visible. The composites should be hidden (same as current behavior).

**Staging preview:**
During generation staging with `isolatedStagingPreview`, only raster layers should be visible. The composites need to only include raster layer content.

**Disabled entities:**
Disabled entities are skipped during flattening (not rendered into composites).

**Entity type visibility:**
If a type is globally hidden (e.g., all control layers hidden), those entities are excluded from composites.

## Files to Modify

| File | Change |
| ----------------------------------------------- | ------------------------------------------------------- |
| `konva/CanvasLayerFlatteningModule.ts` | **NEW** — Core flattening logic |
| `konva/CanvasManager.ts` | Register new module, integrate into lifecycle |
| `konva/CanvasEntity/CanvasEntityAdapterBase.ts` | Don't auto-add layer to stage; expose attach/detach API |
| `konva/CanvasEntityRendererModule.ts` | Delegate layer arrangement to flattening module |
| `konva/CanvasCompositorModule.ts` | Reuse/share compositing utilities |
| `konva/CanvasStageModule.ts` | No change needed (addLayer/stage management stays) |

## Performance Expectations

| Metric | Before | After |
| ---------------------- | ---------------------------- | -------------------------------------------------- |
| Canvas elements in DOM | N + 2 (background + preview) | 5 (background + behind + active + ahead + preview) |
| Browser composite cost | O(N) per frame | O(1) per frame |
| Layer switch cost | O(1) | O(N) one-time re-flatten |
| Active layer edit cost | O(1) | O(1) unchanged |

The trade-off is that switching the selected entity requires a one-time re-flatten, but this can be made fast with caching and incremental updates. The per-frame rendering cost drops from O(N) to O(1), which is the dominant performance factor.

## Risks and Mitigations

1. **Visual fidelity**: Flattened composites must exactly match the per-layer rendering. Use the same compositing pipeline (`getCompositeCanvas`) to ensure consistency.

2. **Blend mode accuracy**: CSS `mix-blend-mode` on individual canvas elements may differ slightly from `globalCompositeOperation` during canvas compositing. Test thoroughly with all blend modes.

3. **Re-flatten latency**: For 50+ layers with complex content, flattening may take 50-100ms. Mitigate with:
- Async flattening in a Web Worker (see README: "Perf: Konva in a web worker")
- Show a brief transition indicator during re-flatten
- Incremental flattening (only re-render changed entities)

4. **Memory**: Two extra composite canvases at full resolution. For a 4K canvas, this is ~32MB per composite. Acceptable for modern systems.

## Dependencies

- Konva's `layer.toCanvas()` or manual canvas rendering via `getCanvas()` on each adapter
- `CanvasCompositorModule` compositing utilities (hash computation, canvas compositing)
- `CanvasCacheModule` for caching flattened results

## References

- Current compositing: `konva/CanvasCompositorModule.ts` lines 204-247
- README future enhancement: `controlLayers/README.md` lines 196-206
- Entity adapter rendering: `konva/CanvasEntity/CanvasEntityAdapterBase.ts`
- Entity z-order management: `konva/CanvasEntityRendererModule.ts` lines 105-146
Loading
Loading