What is Eagle?

Eagle is HTC's AI-powered smart eyewear platform. At its core is Vive — an AI assistant that responds to voice queries, analyzes visual context through the device camera, and surfaces relevant information hands-free. The system spans 20+ distinct interaction modes: from identifying plants, animals, and landmarks, to translating foreign text, reviewing nearby restaurants, and summarizing news — all delivered as brief, spoken responses optimized for an audio-first wearable interface.

As Principal Designer, I owned the complete AI prompt architecture solo — working closely with engineering to implement and refine, and with a PM to prioritize features. The product shipped in 2025, with Taiwan as the primary launch market, which introduced significant multilingual complexity from day one.

On a screenless wearable, every AI response competes with the physical world for attention. Clarity isn't aesthetic — it's functional. The design challenge was building a system that behaves consistently across dozens of distinct interaction types, seven languages, and four different underlying LLMs, while outputting responses that work when spoken aloud.

How the system routes queries

Eagle AI Middleware classifies every query before dispatching to the appropriate prompt class and external service. Voice input, button-triggered actions, and automatic photo detection each enter the pipeline differently — routed to one of four branches: Phone Assistant, Glasses Action, Verbal Query, or Visual Query.

The architecture distinguishes between two prompt types with fundamentally different behaviors. Chat prompts maintain conversation history for stateful interactions. Plugin prompts are stateless — essential for all image-based queries where prior context doesn't carry over.

A repeatable engineering methodology

Every prompt was built from a standardized anatomy — ensuring behavioral consistency across 20+ classes and enabling systematic iteration. The anatomy was applied regardless of prompt type, with optional components added as the interaction required.

Prompt taxonomy

Each prompt class handles a distinct interaction type with its own input modality and AI capability requirements.

Multi-stage prompt pipelines

Two interaction types required chained prompt architectures — where one prompt's output becomes the next prompt's input. This improved accuracy over single-prompt approaches by separating classification from response generation.

Ingredients → Recipe: A first prompt identifies and lists all visible ingredients from the camera image, outputting a structured ingredient list. A second grounding prompt receives that list and suggests dishes using real-time recipe sources. Splitting the task improved ingredient identification accuracy significantly over a combined approach.

Signboard → Restaurant (B8/C9): A signboard detection prompt extracts the store name via OCR and outputs a structured Type/Text/Language schema. That name is passed to a Places API lookup, whose results feed a restaurant review prompt (C9). Three sequential stages, two engineered prompt classes.

The B8 challenge

B8 was the most complex prompt in the system — a monolithic chain designed to handle any image that didn't fit a preset category. It had to execute three sequential behaviors in one prompt: signboard detection → landmark identification → general scene description, with conditional fallthrough between each stage.

The primary engineering challenges were chain-breaking (the model responding at Stage 1 even when no sign was present), output schema enforcement (returning plain text instead of the required structured format), and false landmark detection (identifying non-landmark buildings as famous ones). Each required distinct prompt interventions — explicit STOP instructions, numbered template sections with concrete examples, and location-aware constraints tying identification to the user's GPS proximity.

As Gemini improved from Flash 2 → 2.5 → 3, B8 became progressively more stable — demonstrating that monolithic chaining is viable when the underlying model is capable enough, but complex chaining ultimately belongs in system architecture rather than a single prompt.

Designing for the camera as an input

The visual query family is the most complex subsystem — nine prompt classes sharing a common input (the device camera) but requiring distinct recognition capabilities, behavioral rules, and failure states. Each was tested against a broad set of images including deliberately empty frames to verify failure handling.

Pointing gesture as an interaction primitive

A deliberate design decision: the system detects whether a user's index finger is pointing at something in the camera frame and responds to the pointed object, not the general scene. This required precise prompt engineering — not just enabling the capability but making it reliable.

The detection logic required: an index finger extended with a clear directional line from fingertip to a specific object, with other fingers curled or closed. Cropped fingers, hands merely present without direction, and ambiguous gestures are explicitly not treated as pointing. When pointing is detected the response leads with acknowledgment and addresses the pointed subject specifically. When ambiguous, the system falls back to general scene description — ensuring no false positive attributions.

Formatting as a functional requirement

Every response is spoken aloud on a wearable device. Output formatting constraints aren't stylistic — they're functional requirements for audio delivery. Each rule below was implemented across all 12+ prompt classes.

Iteration process

All prompts were evaluated on Google Vertex AI before deployment — not just in conversation. The process followed a define → write → test → evaluate → refine → ship loop, with the feedback arrow returning from testing directly to writing. Prompt section ordering was treated as a design variable — moving constraints before or after instructions often changed model behavior meaningfully.

Issue tracking

Every post-ship issue was logged in the PQCS system. Hundreds of tickets were triaged, diagnosed, and resolved — with prompt changes tracked against specific issue numbers. The primary issue categories were:

Sample of logged issues from the Eagle PQCS quality tracking system. Color indicates severity — red: critical, yellow: in progress, green: resolved.

Systemic challenges & mitigations

Multi-LLM feature evaluation

Prompts were evaluated across four LLMs to assess portability and capability coverage. Gemini Flash 2 and ChatGPT 4.1 were selected as production models based on the results.

A meta-prompt system

The Custom Prompt Builder is a prompt that writes prompts. A user speaks a raw, casual instruction — "tell me about any zoo I'm near" — and the system converts it into a fully structured, safety-aware, language-compliant AI prompt ready for downstream processing. This lets Eagle extend its own AI capabilities beyond preset prompt classes without requiring engineering intervention.

The system uses a two-stage pipeline: a Refiner (25-word ceiling) compresses and clarifies the user's intent, then a Composer builds the full structured prompt with numbered instructions, safety boundaries, language constraints, and photo context. Whether a photo is required changes the entire prompt structure — the builder detects this and adds or removes visual analysis steps accordingly.

Iteration arc — 8 documented versions

Schema design & structured output

The Meeting Summary feature required designing both the prompt architecture and the JSON output schema — working from an engineering-provided spec and iterating toward a structure that served three distinct display modes: Detailed, Default, and Concise.

The engineering schema provided flat keypoints with raw quote arrays. My redesign introduced hierarchical structure, word limits per field, speaker attribution with timestamp formatting, and the insight layer — renamed from "quotes" to allow supporting points that aren't always direct quotations. A subtitle field was added in v8 to give each summary richer context beyond its subject line.

A key technical discovery: setting max output tokens to 500–1000 truncated the JSON response. The budget needed to cover input transcript, output JSON, and schema overhead simultaneously — requiring a minimum of 1500 tokens to function reliably.

What this project taught me