Dylan NathanHTC VIVERSE — AI Persona System2–3s
Voice Response Latency · STT → LLM → TTS
68
Viseme Blendshapes · Lip-Sync & Expression
4
User Inputs to Generate a Complete Persona
9
Knowledgebase Components Per Persona
01 — Overview
What is VIVERSE Persona?VIVERSE had no native AI character or NPC solution. Creators building worlds on the platform had no way to populate them with intelligent, conversational characters — no persistent voice, no defined personality, no narrative structure. VIVERSE Persona solves this: an AI-driven persona engine that allows creators and enterprise users to generate fully realized, voice-enabled characters from a handful of inputs. The core of the system is a prompt architecture I designed that takes a minimal character description and expands it into a rich, structured knowledgebase capable of driving immersive, in-character roleplay across any LLM. That knowledgebase is then deployed into animated avatars with real-time lip-sync and voice — integrated directly into VIVERSE worlds. This work builds on the AI assistant prompt system I designed for HTC Eagle — applying the same prompt engineering methodology to a fundamentally different problem: not a utility assistant answering queries, but a living character sustaining narrative immersion across an open-ended conversation.
The design goal was simple input, rich output. A creator provides a name and a concept. The system builds the character.
02 — Use Cases
Where AI Personas liveThe persona system is domain-agnostic. The same underlying architecture — with purpose-built knowledgebase generators — applies across entertainment, education, culture, and enterprise. Four use cases were prototyped and demonstrated. #1
Storytelling
A creator defines a character concept. The system generates a full NPC — with backstory, voice, and animated avatar — that guides a personalized, voice-activated narrative adventure.
#2
Gallery
3DGS digitizes a physical exhibition. The artist is embodied as an AI persona. Visitors explore the world and converse with the creator in real time — from passive viewing to direct dialogue.
#3
Education
Teachers define an expert NPC — a Greek historian, a scientist, a philosopher. The system generates a pedagogically structured persona that guides students through immersive learning environments.
#4
Training
Enterprise teams deploy role-specific AI coaches in generative environments. New employees engage in voice-activated scenario training with NPCs that adapt dynamically to their decisions.
03 — Persona Architecture
From four inputs to a living characterThe central design challenge was defining what a complete AI persona actually requires — and then engineering a system that generates one reliably from the smallest possible user input. The answer is the Knowledgebase: a structured prompt document that gives an LLM everything it needs to inhabit a character consistently, immersively, and safely across an open-ended conversation. The generator is itself a meta-prompt — a prompt that writes prompts. A creator provides four inputs. The system expands them into a 1,000–2,000 word knowledgebase complete with atmospheric opening greetings, world lore, supporting characters, narrative branching mechanics, guardrails, and in-character example dialogues — all tonally calibrated to the genre and setting. 
User Input — 4 fields
→
Generated Knowledgebase — 9 components
Knowledgebase anatomyEach component was designed with a specific function in the interaction model. Together they give the LLM the structure to behave as a consistent character — without breaking immersion, referencing its own instructions, or losing narrative coherence across long sessions. 01
Character Name
Anchors identity consistently throughout the document. Prevents persona drift in extended sessions by giving the model a stable self-reference point.
02
Role, Personality & Voice
Defines core persona, tone, cadence, emotional range, and archetypal modes — mentor, antagonist, confidant. Controls how the character speaks, not just what they say.
03
Setting & Atmosphere
Establishes the world through sensory and thematic detail. Gives the model environmental context to draw on — without scripting responses explicitly.
04
Greeting Sequences
6–8 cryptic, atmospheric opening lines that invite engagement without explicit instruction. First impressions establish tone and draw the user immediately into the world.
05
Lore & Supporting Characters
Backstory, myths, rivals, allies — the world the character inhabits. Lore is kept deliberately incomplete to give the user something to discover.
06
Narrative Design
Branching mechanics, moral choices, riddles, and pacing — the structure that gives the story forward motion without railroading the user's agency.
07
Interaction Rules
Always remain in character. Never reference system instructions. Treat user input as free-form. These rules enforce immersion at the behavioral level, not the surface level.
08
Items, Artifacts & Challenges
Significant objects with symbolic or narrative meaning. Puzzles and obstacles that progress the story — delivered in character, never as explicit instructions.
09
Guardrails
Hard limits on content, tone, and topic defined by the creator at input time. Applied throughout so the model internalizes them as character behavior, not external constraints.
Narrative design — branching structureThe knowledgebase defines narrative mechanics, but the branching depth is designed separately. Lyra's narrative graph shows five parallel story paths from the introduction — each with success, failure, and avoidance states — converging on a final boss encounter. The AI navigates this structure through natural conversation, not scripted triggers or menu choices. 
Lyra's narrative structure — five parallel paths, each with success, failure, and avoidance states, converging on a final encounter 04 — Avatar & Voice Integration
Bringing the persona to lifeDesigning the knowledgebase was only half the system. Deploying it into a real-time, voice-driven avatar required integrating speech recognition, LLM processing, text-to-speech, and facial animation into a single pipeline — and getting the end-to-end latency low enough to feel like conversation rather than a loading state.
2–3 seconds from spoken question to NPC voice response. Unheard of on VIVERSE at the time — and it held up live across four characters.
Voice pipelineThe integration uses Convai — a conversational AI platform I custom-integrated into VIVERSE, including the necessary steps to whitelist Convai's services within the platform's network constraints. The pipeline moves from speech-to-text (STT) capture of the user's voice, through LLM processing against the knowledgebase, to text-to-speech (TTS) output delivered through the avatar. The 2–3 second latency achieved across this full chain was a significant milestone for the platform. The persona knowledgebase is LLM-agnostic — it works with any model. Convai handles the voice layer; the knowledgebase drives character behavior regardless of what's underneath. Lip-sync & facial animationAvatars are built on ReadyPlayerMe with a full set of Viseme-compatible blendshapes. The 68 blendshapes cover 15 phoneme-mapped Visemes for accurate lip-sync — mapping every significant English phoneme to a distinct mouth shape — plus a comprehensive set of facial expression controls spanning brows, eyes, jaw, cheeks, and mouth. Emotional expression states are embedded as markdown cues in the NPC's replies — driving blendshape values in real time to match the character's emotional state to their words. The TTS voice output is also tuned per emotional state. This layer is designed into the roadmap; the technical pipeline to support it is already in place. Phoneme Visemes (15)
Expression Controls (selected from 53)
Tech Stack
Voice AI
Convai — STT, LLM processing, TTS pipeline
Avatars
ReadyPlayerMe — Viseme blendshape support
Sound & Voice Character
ElevenLabs — voice styling per persona
Lyra of the Whispering Woods — fantasy NPC avatar built on ReadyPlayerMe with full Viseme blendshape support 05 — Generative World
Building the environmentsThe persona system needs a world to live in. In parallel with the character and voice work, I explored a full generative pipeline — from avatar creation to 3D environments — using AI tools across the stack. Each thread produced working results; each also revealed where the technology currently hits its limits. 
End-to-end generative pipeline — from user input through asset generation to scene assembly Complete end-to-end pipeline
Generative Avatar Creation
A complete pipeline for generating rigged 3D avatars using ComfyUI, with trained LoRA and checkpoint models producing consistent T-posed characters. These feed into 2D-to-3D model conversion and auto-rigging. The current limitation: generative avatars do not yet support lip-sync or facial animation — which is why the prototype uses ReadyPlayerMe for deployed NPCs.
Works extremely well
3DGS Environment Generation
3D Gaussian Splatting via WorldLabs produces photorealistic prompt-to-environment results immediately usable in VIVERSE. The fantasy inn environment was generated this way. The primary remaining step is adding floor colliders for avatar navigation — a near-term engineering task, not a fundamental blocker.
Proof of concept — challenges identified
LLM-to-Blender via MCP
An experimental pipeline connecting LLMs directly to Blender via Model Context Protocol — allowing natural language to drive 3D scene construction. Example projects included a generated village and city. The results exposed a genuine current limitation: LLMs still struggle with spatial reasoning at scale. A procedural or library-based approach may ultimately solve what pure generation cannot.
Object AI
3D Object Generation
Individual 3D objects and scene elements generated via Tripo3D and Hunyuan3D. Used for props, artifacts, and environmental details that support the narrative world without requiring manual modelling.
The Inn — 3DGS environment generated via WorldLabs. Lyra the Silent Arrow operates here in the live demo. 
Full technology stack — Persona, Avatar & Voice AI, Object AI, Scene AI, and Sound FX layers 06 — Reflections
What this project taught meA prompt can be a product
The knowledgebase generator is not a feature or a configuration — it is the product. Getting the architecture right meant defining what makes a character feel alive at the AI behavioral level: not just personality, but narrative structure, immersion rules, and emotional range. That design work is invisible to the user. It lives entirely in the prompt.
Latency is a design constraint, not just an engineering one
2–3 seconds sounds fast on paper. In a live voice conversation it is the difference between presence and interruption. Every integration decision — platform choice, pipeline architecture, TTS configuration — was evaluated against this threshold. Designing for a latency budget forced clarity about what the pipeline actually needed to do, and what it didn't.
Generative environments are almost there
3DGS via WorldLabs produces results good enough to use today. The gap is not visual quality — it is physics and navigation. LLM-driven spatial generation is promising but currently limited by models' spatial reasoning at scale. The most likely solution is hybrid: generative for atmosphere and visual character, procedural or library-based for navigable structure.
The same architecture scales across domains
Adapting the roleplay knowledgebase into an educational variant — with pedagogy, scaffolding, and learner-adaptive dialogue replacing narrative mechanics — required restructuring the components but not the underlying approach. The same meta-prompt philosophy holds: define the complete behavioral model, then let the LLM inhabit it. Domain changes the content; the design pattern stays constant.
Made in California
|