Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion docs/design/56-focused-npc-dialogue-portrait-lipsync-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,42 @@ blendshape and bone effectors. It also supports preset or custom effector lists.
This matches our tier 3 direction, but it should remain behind the isolated
Convai import lane and not be required for the first focused dialogue UI pass.

Ida Faber's blendshape documentation is especially relevant for semi-real
candidate bodies. It documents Apple ARKit 52 blendshapes, facial capture, and
body customization morphs, which makes Ida-compatible bodies plausible targets
for an ARKit-style lip sync profile. The implementation should still inspect
the actual imported Unity mesh before assuming every purchased Ida character has
the same available blendshape names.

Reference:

- [Convai Unity: Adding Lip-Sync to your Character](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin/adding-lip-sync-to-your-character)
- [Ida Faber Docs: Blendshapes and Facial Animation](https://docs.idafaber3d.com/features/blendshapes)

### Ida Faber ARKit Blendshape Lane

Use this lane only for Ida-family bodies whose imported Unity meshes expose the
needed face blendshapes.

Implementation rules:

- Add an `IdaFaberArkit52` lip sync profile only after inspecting the real
imported `SkinnedMeshRenderer` blendshape names.
- Start with a minimal mouth set for alpha: `jawOpen`, mouth close or funnel
shapes if present, and optional blink shapes.
- Keep the text-timed fallback available for any Ida body missing compatible
facial targets.
- Do not drive full expression changes from dialogue text without authored
emotion rules. Mouth motion and blink are enough for the first pass.
- Treat ARKit blendshape weights as local presentation data only.

Evidence:

- Screenshot or inspector note listing the focused Ida body face mesh and its
available mouth/blink blendshape names.
- Play Mode capture showing text-timed or audio-driven mouth movement on one
compatible Ida body.
- Fallback note for one body or prefab without compatible face blendshapes.

---

Expand Down Expand Up @@ -294,6 +327,8 @@ Build:
- Do not require Convai to be imported for D1 or D2.
- When Convai is imported later, map compatible bodies through a
character-specific lip sync profile instead of one global face assumption.
- For Ida-family bodies, validate the real Unity blendshape list first and use
an ARKit-style profile only when the imported mesh supports it.

Evidence:

Expand Down Expand Up @@ -355,4 +390,3 @@ Do not cut:
- speaking-state cleanup on exit or failure
- server-authority boundary
- readable focused dialogue layout

Loading