I built Rena for the Gemini Live Agent Challenge 2026 — a personal AI health companion where you never have to open a logging screen. You just talk.

Try it: rena-490107-f0f28.web.app · Code: github.com/aag1091-alt/Rena · Demo: Watch on YouTube

The idea

Most health apps fail because tracking is a chore. You have to open the app, find the right screen, tap in numbers — dozens of times a day. Rena flips this with voice-first interaction powered by Gemini Live:

  • “I had salmon, rice, and sparkling water for lunch”
  • “Add bodyweight squats to my workout plan”
  • “Plan my meals for tomorrow — I have a long run in the morning”

Rena listens, understands, and acts. The app also starts with one question: “What are you working toward?” You tell her in your own words — “I want to feel confident at my friend’s wedding in July”, “Beach trip in 10 weeks” — and every recommendation she makes is filtered through that specific goal, not a generic calorie number.

The app

Four tabs, each with a dedicated voice context:

  • Home — goal countdown, daily calorie breakdown, protein/water progress, AI morning insight
  • History — scrollable day log with macros, workouts, and a Gemini-generated daily insight
  • Plan — workout plan + meal plan, generated through voice conversation with Rena
  • Scan — camera or gallery scan; Rena identifies food and estimates calories per item

Technologies

AI & ML

Technology How it’s used
Gemini Live API Real-time bidirectional voice — the core of all voice interaction
Gemini 2.5 Flash (gemini-2.5-flash-native-audio-latest) Agent reasoning, plan generation, day insights, coaching scripts
Gemini Flash Vision Food photo recognition — identifies items, returns nutrition estimates
Veo 2 AI-generated exercise demonstration videos
Imagen Vision board image generation on goal setup
Google ADK Agent framework — tools, routing, session management
Google Cloud TTS (en-US-Neural2-F) Rena’s coaching voiceover for exercise videos

Infrastructure

Service Role
Cloud Run Hosts the Rena agent (min 1 instance — no cold-start voice drops)
Firestore All user data — logs, plans, goals, prompts, insights, video jobs
Cloud Storage Exercise videos + vision board images
Firebase Hosting PWA web companion
Cloud Build CI/CD — GitHub push triggers build + deploy

Application stack

  • iOS: SwiftUI, AVAudioEngine, URLSessionWebSocketTask
  • Web PWA: Vanilla JS, Web Audio API + AudioWorklet, WebSocket, Service Worker
  • Backend: Python, FastAPI, Google ADK

Architecture

Rena's architecture: app, agent on Cloud Run, and the services behind it The iOS app or web PWA, with Home, History, Plan and Scan tabs, talks to the Rena agent on Cloud Run over a WebSocket for audio and REST for data. Inside the agent, the Google ADK agent core exposes tools: log_meal, log_water, log_workout, log_weight, scan_image, generate_workout_plan, generate_meal_plan, save_tomorrow_plan_note and get_recent_workouts. A context prompt system keeps prompts in Firestore so they can be edited without a redeploy, injects the RENA MEMORY block at every session start, has per-tab contexts for home, history, scan, plan and goal, and sends tool_status WebSocket messages for live save indicators. The agent uses three services: Firestore for all data; Gemini APIs (Live, Flash, Vision, Veo 2, Imagen and TTS); and Cloud Storage with exercise_videos/ and vision_journey/ folders. iOS app / web PWA Home · History · Plan · Scan WebSocket audio REST data Rena agent · Cloud Run Google ADK agent core log_meal · log_water log_workout · log_weight scan_image generate_workout_plan generate_meal_plan save_tomorrow_plan_note get_recent_workouts Context prompt system Prompts in Firestore, editable, no redeploy [RENA MEMORY] injected at session start Per-tab: home, history, scan, plan, goal tool_status → live save indicators Firestore all data Gemini APIs Live · Flash · Vision Veo 2 · Imagen · TTS Cloud Storage exercise_videos/ vision_journey/
One agent on Cloud Run holds the tools and the prompts; everything it stores or generates lives in three services behind it.

Voice session data flow:

One Rena voice session, from tap to reply The user taps Rena, which opens a WebSocket to /ws/{user_id} with the current tab as context and the user's timezone. The backend fetches the RENA MEMORY block and the prompt for that tab. Microphone audio is streamed as 16 kHz PCM over the WebSocket into Google ADK and Gemini Live, which works out the intent and calls a tool such as log_meal, log_workout or scan_image. Before the tool runs, a tool_status message such as "Logging your meal…" shows a live save indicator, and the tool writes to Firestore. Rena's reply is streamed back as 24 kHz audio chunks and played via AVAudioEngine. You tap Rena /ws/{user_id}?context={tab}&tz=… Backend loads context [RENA MEMORY] + that tab's prompt PCM audio · 16 kHz ADK → Gemini Live intent → tool called log_meal · log_workout · scan_image tool_status “Logging your meal…” live save indicator Tool writes to Firestore Rena replies out loud 24 kHz audio chunks · AVAudioEngine
One tap, one round trip: the context is loaded before you speak, and the save indicator shows up before the tool runs.

The exercise video pipeline

One of the more ambitious features: AI-generated coaching videos for workout exercises, generated on demand and cached forever.

1. Gemini 2.5 Flash writes a coaching script
   (setup cues, movement feel, breath, safety — BLOCK_NONE filters
    so anatomical terms pass through)

2. Random trainer gender picked for visual variety

3. Veo 2 job submitted with exercise + muscles + script as direction
   "No text, subtitles, captions or overlays on screen"
   → returns job_id immediately; stored in Firestore

4. iOS/web polls every 5s for status

5. Google Cloud TTS (en-US-Neural2-F) generates coaching voiceover
   We tried matching a separate trainer voice to the video — the timing
   never lined up. The fix: use Rena's own voice throughout.
   Same voice the user has been talking to all along.

6. ffmpeg muxes Veo video + TTS audio into a single .mp4

7. Uploaded to GCS — same exercise never regenerates

8. AVQueuePlayer + AVPlayerLooper plays seamlessly on iOS

Exercises with AI-generated videos available today: Bodyweight Squats, Plank, Walking Lunges, Elliptical Trainer.

Key challenges

1. Getting Gemini Live to behave inside ADK

The hardest part was making the voice agent reliable rather than just impressive in a demo. Gemini Live tends to interrupt itself, end turns prematurely, or stop responding after tool calls.

The solution was treating prompts like software — versioned, tested, iterated. Every context (home logging, plan generation, history editing, scan) has its own system prompt stored in Firestore. Live-editable without a redeploy, meaning prompt changes could be tested in the running app within seconds.

Key prompt techniques:

  • [RENA MEMORY] block injected at session start — goal, today’s calories, recent meals, workouts, weight trend — so Rena never has to ask what you’ve already told her
  • Explicit post-tool-call instructions: “after saving, tell the user what you logged in one sentence and wait”
  • thinking_budget=0 via a monkey-patch on Gemini.connect() — ADK’s standard config path silently dropped thinking_config before it reached the Live API

2. Real-time save indicators

When Rena calls a tool there’s a gap — the user doesn’t know if something is happening or if the session stalled. Before each tool runs, the backend sends {"type": "tool_status", "message": "Logging your meal…"} over the WebSocket. The voice overlay immediately updates in real time — users see “Building your workout plan…” instead of silence.

3. iOS audio engine continuity

AVAudioEngine is notoriously fragile — audio interruptions can destroy it, and restarting introduces latency. We keep the engine running continuously between sessions, routing PCM frames through an AVAudioMixerNode tap rather than restarting on every session open.

4. Veo 2 audio matching

First approach: record a trainer’s voice reading the coaching script and mux with the Veo video. The timing never matched — the trainer audio was recorded independently of how the generated video moved.

Fix: use Rena’s own TTS voice (en-US-Neural2-F) for the coaching audio. Unexpected upside — users are already familiar with Rena’s voice, so hearing her coach them through an exercise feels natural and consistent.

5. Timezone-aware context

“What did I eat today?” is hard when users are in different timezones and the server runs in UTC. We added tz to the WebSocket URL, cached it per session, and inject [CURRENT_LOCAL_TIME] into every prompt so Rena always knows the user’s local date.

6. Prompt iteration speed

8+ contexts required dozens of prompt iterations each. The Firestore-backed system meant zero redeploys. A seed_prompts.py script keeps prompts in version control and syncs to Firestore on demand.

Try it yourself

Web app (instant, no install): rena-490107-f0f28.web.app

  1. Open in Chrome or Safari on your phone
  2. Sign in with Google and complete the quick onboarding
  3. Tap ⚙ Settings → Seed 7 Days Data to pre-populate with realistic history
  4. Tap the Rena orb (center tab), allow microphone, and start talking

The web app is optimised for mobile screen sizes. If you’re on desktop, open DevTools and switch to iPhone view.

iOS app: Clone github.com/aag1091-alt/Rena, open ios/Rena/Rena.xcodeproj in Xcode, run on a physical iPhone. The app connects to the live Cloud Run backend — no local server needed.

What’s next

  • Push notifications for morning nudges
  • Gallery scan — Rena proactively finds food photos you already took
  • Pattern filling — if nothing is logged by noon, Rena asks about your usual routine
  • Streak milestones with Imagen-evolved vision boards
  • watchOS for passive tracking

App screenshots

Home

Home

History

History

Plan

Plan

Scan

Scan

Built with Google Cloud, Gemini Live API, Veo 2, Imagen, and Google ADK for the Gemini Live Agent Challenge 2026.

Another take on streaming live audio from a phone: near real-time transcription with faster-whisper on my own server.

Rena’s need to remember users across sessions is where VaultMem started — an encrypted memory library for AI agents.