# Building Rena — An AI Health Companion with Gemini Live, Veo 2, and Google ADK

By Avinash Gosavi · 2026-03-16 · https://www.avinashgosavi.com/post/building-rena-ai-health-companion-gemini-live/
Topics: Agents, Voice AI

> How I built Rena for the Gemini Live Agent Challenge 2026 — a voice-first health companion using Gemini Live, Veo 2 and Google ADK.

---

I built Rena for the **Gemini Live Agent Challenge 2026** — a personal AI health companion where you never have to open a logging screen. You just talk.

> **Try it:** [rena-490107-f0f28.web.app](https://rena-490107-f0f28.web.app) · **Code:** [github.com/aag1091-alt/Rena](https://github.com/aag1091-alt/Rena) · **Demo:** [Watch on YouTube](https://www.youtube.com/watch?v=f-ZZw0CTivU)

<div style="position:relative;padding-bottom:56.25%;height:0;overflow:hidden;border-radius:12px;margin:1.5rem 0;">
  <iframe src="https://www.youtube.com/embed/f-ZZw0CTivU" title="Rena demo" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen style="position:absolute;top:0;left:0;width:100%;height:100%;border:0;border-radius:12px;"></iframe>
</div>

---

## The idea

Most health apps fail because tracking is a chore. You have to open the app, find the right screen, tap in numbers — dozens of times a day. Rena flips this with voice-first interaction powered by Gemini Live:

- *"I had salmon, rice, and sparkling water for lunch"*
- *"Add bodyweight squats to my workout plan"*
- *"Plan my meals for tomorrow — I have a long run in the morning"*

Rena listens, understands, and acts. The app also starts with one question: *"What are you working toward?"* You tell her in your own words — *"I want to feel confident at my friend's wedding in July"*, *"Beach trip in 10 weeks"* — and every recommendation she makes is filtered through that specific goal, not a generic calorie number.

---

## The app

Four tabs, each with a dedicated voice context:

- **Home** — goal countdown, daily calorie breakdown, protein/water progress, AI morning insight
- **History** — scrollable day log with macros, workouts, and a Gemini-generated daily insight
- **Plan** — workout plan + meal plan, generated through voice conversation with Rena
- **Scan** — camera or gallery scan; Rena identifies food and estimates calories per item

---

## Technologies

### AI & ML

| Technology | How it's used |
|---|---|
| **Gemini Live API** | Real-time bidirectional voice — the core of all voice interaction |
| **Gemini 2.5 Flash** (`gemini-2.5-flash-native-audio-latest`) | Agent reasoning, plan generation, day insights, coaching scripts |
| **Gemini Flash Vision** | Food photo recognition — identifies items, returns nutrition estimates |
| **Veo 2** | AI-generated exercise demonstration videos |
| **Imagen** | Vision board image generation on goal setup |
| **Google ADK** | Agent framework — tools, routing, session management |
| **Google Cloud TTS** (`en-US-Neural2-F`) | Rena's coaching voiceover for exercise videos |

### Infrastructure

| Service | Role |
|---|---|
| **Cloud Run** | Hosts the Rena agent (min 1 instance — no cold-start voice drops) |
| **Firestore** | All user data — logs, plans, goals, prompts, insights, video jobs |
| **Cloud Storage** | Exercise videos + vision board images |
| **Firebase Hosting** | PWA web companion |
| **Cloud Build** | CI/CD — GitHub push triggers build + deploy |

### Application stack

- **iOS:** SwiftUI, AVAudioEngine, URLSessionWebSocketTask
- **Web PWA:** Vanilla JS, Web Audio API + AudioWorklet, WebSocket, Service Worker
- **Backend:** Python, FastAPI, Google ADK

---

## Architecture

> **Diagram: Rena's architecture: app, agent on Cloud Run, and the services behind it** ([see it in the post](https://www.avinashgosavi.com/post/building-rena-ai-health-companion-gemini-live/))
>
> The iOS app or web PWA, with Home, History, Plan and Scan tabs, talks to the Rena agent on Cloud Run over a WebSocket for audio and REST for data. Inside the agent, the Google ADK agent core exposes tools: log_meal, log_water, log_workout, log_weight, scan_image, generate_workout_plan, generate_meal_plan, save_tomorrow_plan_note and get_recent_workouts. A context prompt system keeps prompts in Firestore so they can be edited without a redeploy, injects the RENA MEMORY block at every session start, has per-tab contexts for home, history, scan, plan and goal, and sends tool_status WebSocket messages for live save indicators. The agent uses three services: Firestore for all data; Gemini APIs (Live, Flash, Vision, Veo 2, Imagen and TTS); and Cloud Storage with exercise_videos/ and vision_journey/ folders.
>
> *One agent on Cloud Run holds the tools and the prompts; everything it stores or generates lives in three services behind it.*

**Voice session data flow:**

> **Diagram: One Rena voice session, from tap to reply** ([see it in the post](https://www.avinashgosavi.com/post/building-rena-ai-health-companion-gemini-live/))
>
> The user taps Rena, which opens a WebSocket to /ws/{user_id} with the current tab as context and the user's timezone. The backend fetches the RENA MEMORY block and the prompt for that tab. Microphone audio is streamed as 16 kHz PCM over the WebSocket into Google ADK and Gemini Live, which works out the intent and calls a tool such as log_meal, log_workout or scan_image. Before the tool runs, a tool_status message such as "Logging your meal…" shows a live save indicator, and the tool writes to Firestore. Rena's reply is streamed back as 24 kHz audio chunks and played via AVAudioEngine.
>
> *One tap, one round trip: the context is loaded before you speak, and the save indicator shows up before the tool runs.*

---

## The exercise video pipeline

One of the more ambitious features: AI-generated coaching videos for workout exercises, generated on demand and cached forever.

```
1. Gemini 2.5 Flash writes a coaching script
   (setup cues, movement feel, breath, safety — BLOCK_NONE filters
    so anatomical terms pass through)

2. Random trainer gender picked for visual variety

3. Veo 2 job submitted with exercise + muscles + script as direction
   "No text, subtitles, captions or overlays on screen"
   → returns job_id immediately; stored in Firestore

4. iOS/web polls every 5s for status

5. Google Cloud TTS (en-US-Neural2-F) generates coaching voiceover
   We tried matching a separate trainer voice to the video — the timing
   never lined up. The fix: use Rena's own voice throughout.
   Same voice the user has been talking to all along.

6. ffmpeg muxes Veo video + TTS audio into a single .mp4

7. Uploaded to GCS — same exercise never regenerates

8. AVQueuePlayer + AVPlayerLooper plays seamlessly on iOS
```

Exercises with AI-generated videos available today: Bodyweight Squats, Plank, Walking Lunges, Elliptical Trainer.

---

## Key challenges

### 1. Getting Gemini Live to behave inside ADK

The hardest part was making the voice agent *reliable* rather than just impressive in a demo. Gemini Live tends to interrupt itself, end turns prematurely, or stop responding after tool calls.

The solution was treating prompts like software — versioned, tested, iterated. Every context (home logging, plan generation, history editing, scan) has its own system prompt stored in Firestore. Live-editable without a redeploy, meaning prompt changes could be tested in the running app within seconds.

Key prompt techniques:
- **`[RENA MEMORY]` block** injected at session start — goal, today's calories, recent meals, workouts, weight trend — so Rena never has to ask what you've already told her
- Explicit post-tool-call instructions: *"after saving, tell the user what you logged in one sentence and wait"*
- `thinking_budget=0` via a monkey-patch on `Gemini.connect()` — ADK's standard config path silently dropped `thinking_config` before it reached the Live API

### 2. Real-time save indicators

When Rena calls a tool there's a gap — the user doesn't know if something is happening or if the session stalled. Before each tool runs, the backend sends `{"type": "tool_status", "message": "Logging your meal…"}` over the WebSocket. The voice overlay immediately updates in real time — users see *"Building your workout plan…"* instead of silence.

### 3. iOS audio engine continuity

`AVAudioEngine` is notoriously fragile — audio interruptions can destroy it, and restarting introduces latency. We keep the engine running continuously between sessions, routing PCM frames through an `AVAudioMixerNode` tap rather than restarting on every session open.

### 4. Veo 2 audio matching

First approach: record a trainer's voice reading the coaching script and mux with the Veo video. The timing never matched — the trainer audio was recorded independently of how the generated video moved.

Fix: use Rena's own TTS voice (`en-US-Neural2-F`) for the coaching audio. Unexpected upside — users are already familiar with Rena's voice, so hearing her coach them through an exercise feels natural and consistent.

### 5. Timezone-aware context

*"What did I eat today?"* is hard when users are in different timezones and the server runs in UTC. We added `tz` to the WebSocket URL, cached it per session, and inject `[CURRENT_LOCAL_TIME]` into every prompt so Rena always knows the user's local date.

### 6. Prompt iteration speed

8+ contexts required dozens of prompt iterations each. The Firestore-backed system meant zero redeploys. A `seed_prompts.py` script keeps prompts in version control and syncs to Firestore on demand.

---

## Try it yourself

**Web app (instant, no install):** [rena-490107-f0f28.web.app](https://rena-490107-f0f28.web.app)

1. Open in Chrome or Safari on your phone
2. Sign in with Google and complete the quick onboarding
3. Tap **⚙ Settings → Seed 7 Days Data** to pre-populate with realistic history
4. Tap the **Rena orb** (center tab), allow microphone, and start talking

> The web app is optimised for mobile screen sizes. If you're on desktop, open DevTools and switch to iPhone view.

**iOS app:** Clone [github.com/aag1091-alt/Rena](https://github.com/aag1091-alt/Rena), open `ios/Rena/Rena.xcodeproj` in Xcode, run on a physical iPhone. The app connects to the live Cloud Run backend — no local server needed.

---

## What's next

- Push notifications for morning nudges
- Gallery scan — Rena proactively finds food photos you already took
- Pattern filling — if nothing is logged by noon, Rena asks about your usual routine
- Streak milestones with Imagen-evolved vision boards
- watchOS for passive tracking

---

## App screenshots

<div style="display:flex; gap:12px; justify-content:center; align-items:flex-start; flex-wrap:nowrap; overflow-x:auto;">
  <div style="text-align:center; min-width:0; flex:1;">
    <picture>
      <source type="image/webp" srcset="https://www.avinashgosavi.com/assets/images/rena-home.webp">
      <img width="360" height="778" loading="lazy" decoding="async" src="https://www.avinashgosavi.com/assets/images/rena-home.jpeg" alt="Home" style="width:100%; border-radius:12px; box-shadow:0 2px 10px rgba(0,0,0,0.12);">
    </picture>
    <p style="margin-top:6px; font-size:0.8rem; color:var(--text-muted);">Home</p>
  </div>
  <div style="text-align:center; min-width:0; flex:1;">
    <picture>
      <source type="image/webp" srcset="https://www.avinashgosavi.com/assets/images/rena-history.webp">
      <img width="360" height="778" loading="lazy" decoding="async" src="https://www.avinashgosavi.com/assets/images/rena-history.jpeg" alt="History" style="width:100%; border-radius:12px; box-shadow:0 2px 10px rgba(0,0,0,0.12);">
    </picture>
    <p style="margin-top:6px; font-size:0.8rem; color:var(--text-muted);">History</p>
  </div>
  <div style="text-align:center; min-width:0; flex:1;">
    <picture>
      <source type="image/webp" srcset="https://www.avinashgosavi.com/assets/images/rena-plan.webp">
      <img width="360" height="778" loading="lazy" decoding="async" src="https://www.avinashgosavi.com/assets/images/rena-plan.jpeg" alt="Plan" style="width:100%; border-radius:12px; box-shadow:0 2px 10px rgba(0,0,0,0.12);">
    </picture>
    <p style="margin-top:6px; font-size:0.8rem; color:var(--text-muted);">Plan</p>
  </div>
  <div style="text-align:center; min-width:0; flex:1;">
    <picture>
      <source type="image/webp" srcset="https://www.avinashgosavi.com/assets/images/rena-scan.webp">
      <img width="360" height="778" loading="lazy" decoding="async" src="https://www.avinashgosavi.com/assets/images/rena-scan.jpeg" alt="Scan" style="width:100%; border-radius:12px; box-shadow:0 2px 10px rgba(0,0,0,0.12);">
    </picture>
    <p style="margin-top:6px; font-size:0.8rem; color:var(--text-muted);">Scan</p>
  </div>
</div>

---

*Built with Google Cloud, Gemini Live API, Veo 2, Imagen, and Google ADK for the Gemini Live Agent Challenge 2026.*

*Another take on streaming live audio from a phone: [near real-time transcription with faster-whisper on my own server](https://www.avinashgosavi.com/post/live-audio-transcription-whisper/).*

*Rena's need to remember users across sessions is where [VaultMem](https://www.avinashgosavi.com/post/vaultmem-encrypted-memory-for-ai-agents/) started — an encrypted memory library for AI agents.*

---

Original post: https://www.avinashgosavi.com/post/building-rena-ai-health-companion-gemini-live/