Skip to content

Benchmark role-play model providers behind api.dos.ai #249

@JOY

Description

@JOY

Summary

Benchmark Alibaba Qwen-Character and similar role-play providers behind api.dos.ai as replaceable adapters. This is an R&D task, not an MVP dependency.

Why

SECOND SPAWN needs stronger role-play NPC dialogue, but durable NPC memory, relationship state, PromptTrace, budget, and authority must remain server-owned by Nakama and Fusion. Provider-managed memory must not become canonical game state.

Acceptance Criteria

  • Add an api.dos.ai provider-adapter spike behind a feature flag.
  • Run the same three permanent NPC profiles through current DOS.AI default models and Qwen-Character candidates.
  • Capture latency, response length, cost estimate, structured intent validity, anti-repeat score, hidden-lore violations, and fallback reason.
  • Confirm Unity never calls Alibaba directly and never receives provider keys or provider session tokens.
  • Document whether provider-managed memory is disabled, ignored, or used only as non-canonical short-session cache.
  • Update PromptTrace metadata design if a provider-adapter field is needed.

References

  • docs/design/52-llm-role-play-provider-evaluation.md
  • docs/design/37-ai-npc-backend-client-roadmap.md
  • docs/ARCHITECTURE.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ai-agentOffline player agent, NPC intelligence, and agent observabilityarea:designGame design, economy rules, lore, and GDD workarea:gatewayGo LLM gateway and provider routingpriority:p3Nice to have or latersize:mMedium task

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions