diff --git a/docs/design/56-focused-npc-dialogue-portrait-lipsync-design.md b/docs/design/56-focused-npc-dialogue-portrait-lipsync-design.md index cacfaddd..cb6a6666 100644 --- a/docs/design/56-focused-npc-dialogue-portrait-lipsync-design.md +++ b/docs/design/56-focused-npc-dialogue-portrait-lipsync-design.md @@ -167,9 +167,42 @@ blendshape and bone effectors. It also supports preset or custom effector lists. This matches our tier 3 direction, but it should remain behind the isolated Convai import lane and not be required for the first focused dialogue UI pass. +Ida Faber's blendshape documentation is especially relevant for semi-real +candidate bodies. It documents Apple ARKit 52 blendshapes, facial capture, and +body customization morphs, which makes Ida-compatible bodies plausible targets +for an ARKit-style lip sync profile. The implementation should still inspect +the actual imported Unity mesh before assuming every purchased Ida character has +the same available blendshape names. + Reference: - [Convai Unity: Adding Lip-Sync to your Character](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin/adding-lip-sync-to-your-character) +- [Ida Faber Docs: Blendshapes and Facial Animation](https://docs.idafaber3d.com/features/blendshapes) + +### Ida Faber ARKit Blendshape Lane + +Use this lane only for Ida-family bodies whose imported Unity meshes expose the +needed face blendshapes. + +Implementation rules: + +- Add an `IdaFaberArkit52` lip sync profile only after inspecting the real + imported `SkinnedMeshRenderer` blendshape names. +- Start with a minimal mouth set for alpha: `jawOpen`, mouth close or funnel + shapes if present, and optional blink shapes. +- Keep the text-timed fallback available for any Ida body missing compatible + facial targets. +- Do not drive full expression changes from dialogue text without authored + emotion rules. Mouth motion and blink are enough for the first pass. +- Treat ARKit blendshape weights as local presentation data only. + +Evidence: + +- Screenshot or inspector note listing the focused Ida body face mesh and its + available mouth/blink blendshape names. +- Play Mode capture showing text-timed or audio-driven mouth movement on one + compatible Ida body. +- Fallback note for one body or prefab without compatible face blendshapes. --- @@ -294,6 +327,8 @@ Build: - Do not require Convai to be imported for D1 or D2. - When Convai is imported later, map compatible bodies through a character-specific lip sync profile instead of one global face assumption. +- For Ida-family bodies, validate the real Unity blendshape list first and use + an ARKit-style profile only when the imported mesh supports it. Evidence: @@ -355,4 +390,3 @@ Do not cut: - speaking-state cleanup on exit or failure - server-authority boundary - readable focused dialogue layout -