💡 Inspiration

We wanted to bring open-weights AI into the physical dimension of interactive 3D computing. Traditional chatbots are confined to sterile 2D text windows. With Google’s release of Gemma 4, developers finally have a state-of-the-art, open-weight model with native function calling and multimodal capability that can run locally or via the Gemini API. We built Koko AI—an expressive, full-duplex 3D anime companion that combines 60 FPS WebGL avatar rendering, 0-latency audio-reactive lip-sync, and Gemma 4 agentic tool intelligence.

---

🌟 What It Does

Koko AI is a real-time multimodal 3D VRM companion featuring an energetic, tsundere personality:

• 🧠 Gemma 4 Reasoning & Native Tool Calling: Powered by Google’s Gemma 4 (via llama.cpp local GGUF and Gemini/Mistral API fallbacks), Koko can execute multi-turn agentic tools to interact with your host PC—including running bash commands, managing local files, searching the live web via Tavily, evaluating math, and checking system diagnostics.
• 🎭 60 FPS WebGL VRM Engine: Rendered in Three.js and React Three Fiber with procedural eye-blinking, breathing oscillations, realistic look-at eye tracking, and full OrbitControls camera manipulation.
• 🗣️ 0-Latency Audio-Reactive Lip-Sync: Uses the Web Audio API FFT AnalyserNode to analyze voice frequency energy in real-time, mapping audio spectrums directly to VRM mouth blendshapes (aa, ih, ou, ee, oh).
• ✨ Millisecond-Precise Emotion Synchronization: Synchronizes 6 facial expression presets (Happy, Sad, Angry, Surprised, Relaxed, Neutral) the exact millisecond spoken audio starts.
• 📸 3D Photo Studio (Gallery): Interactive studio to switch poses, control studio lighting, change camera angles, and capture high-resolution transparent screenshots.
• 🪟 Glassmorphic UI & Dark/Light Themes: Modern UI with provider switching, live API key validation, model selectors, and customizable tool toggles.

---

🛠️ How We Built It

• Gemma 4 Agentic Pipeline: Integrated Google’s Gemma 4 with 20-round autonomous tool execution loops. We designed custom tool schemas for bash command execution, file system operations in D:/developer/, real-time web scraping, AST math parsing, and host telemetry.
• Client WebGL Engine: Built with React 19, TypeScript, Vite, @pixiv/three-vrm, @react-three/fiber, and TailwindCSS.
• Full-Duplex Streaming Backend: Python 3.12 WebSocket server utilizing asynchronous binary PCM audio streaming, Faster-Whisper ASR, parallel sentence-level TTS synthesis with synthetic fallback, and summary memory buffer compaction.

---

🥊 Challenges We Ran Into

1. Tool Calling & Anti-Stalling on Open Models: Fine-tuning prompt structures so Gemma 4 reliably emits structured function calling tokens rather than hallucinating answers when tools are required.
2. WebGL Overlay Pointer Interception: Ensuring floating glassmorphic panels and chat overlays didn’t block Three.js OrbitControls mouse drag events (pointer-events: none on overlay roots).
3. Cross-Platform Drive Letter Case Sensitivity: Normalizing Windows drive paths (D:\ vs d:/) to give Gemma 4 unrestricted host execution capabilities without permission bottlenecks.

---

🏆 Accomplishments We’re Proud Of

• True Real-Time Performance: Achieved sub-second audio response times with zero perceived lip-sync lag via client-side Web Audio FFT analysis.
• Native Host Computer Use: Seamlessly connected an open-weights model to host developer environments for real coding and file management tasks.
• 100% Open Source: Fully open-source under the GPLv3 license with a clean modular architecture.

---

📚 What We Learned

• How to harness Gemma 4’s lightweight footprint to execute local function calling alongside high-framerate 3D rendering.
• Best practices in audio-to-blendshape frequency bin mapping for anime-style VRM humanoid avatars.

---

🚀 What’s Next for Koko AI

• Vision understanding pipeline using Gemma 4 multimodal capabilities to let Koko view your screen or webcam live.
• VR headset support via WebXR for full spatial immersion.
• Long-term vector memory embeddings for multi-week persistent companion relationships.