Agents · Avinash Gosavi

Building Rena — An AI Health Companion with Gemini Live, Veo 2, and Google ADK

2026-03-16T00:00:00+00:00

I built Rena for the Gemini Live Agent Challenge 2026 — a personal AI health companion where you never have to open a logging screen. You just talk.

Try it: rena-490107-f0f28.web.app · Code: github.com/aag1091-alt/Rena · Demo: Watch on YouTube

The idea

Most health apps fail because tracking is a chore. You have to open the app, find the right screen, tap in numbers — dozens of times a day. Rena flips this with voice-first interaction powered by Gemini Live:

“I had salmon, rice, and sparkling water for lunch”
“Add bodyweight squats to my workout plan”
“Plan my meals for tomorrow — I have a long run in the morning”

Rena listens, understands, and acts. The app also starts with one question: “What are you working toward?” You tell her in your own words — “I want to feel confident at my friend’s wedding in July”, “Beach trip in 10 weeks” — and every recommendation she makes is filtered through that specific goal, not a generic calorie number.

The app

Four tabs, each with a dedicated voice context:

Home — goal countdown, daily calorie breakdown, protein/water progress, AI morning insight
History — scrollable day log with macros, workouts, and a Gemini-generated daily insight
Plan — workout plan + meal plan, generated through voice conversation with Rena
Scan — camera or gallery scan; Rena identifies food and estimates calories per item

Technologies

AI & ML

Technology	How it’s used
Gemini Live API	Real-time bidirectional voice — the core of all voice interaction
Gemini 2.5 Flash (`gemini-2.5-flash-native-audio-latest`)	Agent reasoning, plan generation, day insights, coaching scripts
Gemini Flash Vision	Food photo recognition — identifies items, returns nutrition estimates
Veo 2	AI-generated exercise demonstration videos
Imagen	Vision board image generation on goal setup
Google ADK	Agent framework — tools, routing, session management
Google Cloud TTS (`en-US-Neural2-F`)	Rena’s coaching voiceover for exercise videos

Infrastructure

Service	Role
Cloud Run	Hosts the Rena agent (min 1 instance — no cold-start voice drops)
Firestore	All user data — logs, plans, goals, prompts, insights, video jobs
Cloud Storage	Exercise videos + vision board images
Firebase Hosting	PWA web companion
Cloud Build	CI/CD — GitHub push triggers build + deploy

Application stack

iOS: SwiftUI, AVAudioEngine, URLSessionWebSocketTask
Web PWA: Vanilla JS, Web Audio API + AudioWorklet, WebSocket, Service Worker
Backend: Python, FastAPI, Google ADK

Architecture

Diagram: Rena's architecture: app, agent on Cloud Run, and the services behind it (see it in the post)
The iOS app or web PWA, with Home, History, Plan and Scan tabs, talks to the Rena agent on Cloud Run over a WebSocket for audio and REST for data. Inside the agent, the Google ADK agent core exposes tools: log_meal, log_water, log_workout, log_weight, scan_image, generate_workout_plan, generate_meal_plan, save_tomorrow_plan_note and get_recent_workouts. A context prompt system keeps prompts in Firestore so they can be edited without a redeploy, injects the RENA MEMORY block at every session start, has per-tab contexts for home, history, scan, plan and goal, and sends tool_status WebSocket messages for live save indicators. The agent uses three services: Firestore for all data; Gemini APIs (Live, Flash, Vision, Veo 2, Imagen and TTS); and Cloud Storage with exercise_videos/ and vision_journey/ folders.
One agent on Cloud Run holds the tools and the prompts; everything it stores or generates lives in three services behind it.

Voice session data flow:

Diagram: One Rena voice session, from tap to reply (see it in the post)
The user taps Rena, which opens a WebSocket to /ws/{user_id} with the current tab as context and the user's timezone. The backend fetches the RENA MEMORY block and the prompt for that tab. Microphone audio is streamed as 16 kHz PCM over the WebSocket into Google ADK and Gemini Live, which works out the intent and calls a tool such as log_meal, log_workout or scan_image. Before the tool runs, a tool_status message such as "Logging your meal…" shows a live save indicator, and the tool writes to Firestore. Rena's reply is streamed back as 24 kHz audio chunks and played via AVAudioEngine.
One tap, one round trip: the context is loaded before you speak, and the save indicator shows up before the tool runs.

The exercise video pipeline

One of the more ambitious features: AI-generated coaching videos for workout exercises, generated on demand and cached forever.

1. Gemini 2.5 Flash writes a coaching script
   (setup cues, movement feel, breath, safety — BLOCK_NONE filters
    so anatomical terms pass through)

2. Random trainer gender picked for visual variety

3. Veo 2 job submitted with exercise + muscles + script as direction
   "No text, subtitles, captions or overlays on screen"
   → returns job_id immediately; stored in Firestore

4. iOS/web polls every 5s for status

5. Google Cloud TTS (en-US-Neural2-F) generates coaching voiceover
   We tried matching a separate trainer voice to the video — the timing
   never lined up. The fix: use Rena's own voice throughout.
   Same voice the user has been talking to all along.

6. ffmpeg muxes Veo video + TTS audio into a single .mp4

7. Uploaded to GCS — same exercise never regenerates

8. AVQueuePlayer + AVPlayerLooper plays seamlessly on iOS

Exercises with AI-generated videos available today: Bodyweight Squats, Plank, Walking Lunges, Elliptical Trainer.

Key challenges

1. Getting Gemini Live to behave inside ADK

The hardest part was making the voice agent reliable rather than just impressive in a demo. Gemini Live tends to interrupt itself, end turns prematurely, or stop responding after tool calls.

The solution was treating prompts like software — versioned, tested, iterated. Every context (home logging, plan generation, history editing, scan) has its own system prompt stored in Firestore. Live-editable without a redeploy, meaning prompt changes could be tested in the running app within seconds.

Key prompt techniques:

[RENA MEMORY] block injected at session start — goal, today’s calories, recent meals, workouts, weight trend — so Rena never has to ask what you’ve already told her
Explicit post-tool-call instructions: “after saving, tell the user what you logged in one sentence and wait”
thinking_budget=0 via a monkey-patch on Gemini.connect() — ADK’s standard config path silently dropped thinking_config before it reached the Live API

2. Real-time save indicators

When Rena calls a tool there’s a gap — the user doesn’t know if something is happening or if the session stalled. Before each tool runs, the backend sends {"type": "tool_status", "message": "Logging your meal…"} over the WebSocket. The voice overlay immediately updates in real time — users see “Building your workout plan…” instead of silence.

3. iOS audio engine continuity

AVAudioEngine is notoriously fragile — audio interruptions can destroy it, and restarting introduces latency. We keep the engine running continuously between sessions, routing PCM frames through an AVAudioMixerNode tap rather than restarting on every session open.

4. Veo 2 audio matching

First approach: record a trainer’s voice reading the coaching script and mux with the Veo video. The timing never matched — the trainer audio was recorded independently of how the generated video moved.

Fix: use Rena’s own TTS voice (en-US-Neural2-F) for the coaching audio. Unexpected upside — users are already familiar with Rena’s voice, so hearing her coach them through an exercise feels natural and consistent.

5. Timezone-aware context

“What did I eat today?” is hard when users are in different timezones and the server runs in UTC. We added tz to the WebSocket URL, cached it per session, and inject [CURRENT_LOCAL_TIME] into every prompt so Rena always knows the user’s local date.

6. Prompt iteration speed

8+ contexts required dozens of prompt iterations each. The Firestore-backed system meant zero redeploys. A seed_prompts.py script keeps prompts in version control and syncs to Firestore on demand.

Try it yourself

Web app (instant, no install): rena-490107-f0f28.web.app

Open in Chrome or Safari on your phone
Sign in with Google and complete the quick onboarding
Tap ⚙ Settings → Seed 7 Days Data to pre-populate with realistic history
Tap the Rena orb (center tab), allow microphone, and start talking

The web app is optimised for mobile screen sizes. If you’re on desktop, open DevTools and switch to iPhone view.

iOS app: Clone github.com/aag1091-alt/Rena, open ios/Rena/Rena.xcodeproj in Xcode, run on a physical iPhone. The app connects to the live Cloud Run backend — no local server needed.

What’s next

Push notifications for morning nudges
Gallery scan — Rena proactively finds food photos you already took
Pattern filling — if nothing is logged by noon, Rena asks about your usual routine
Streak milestones with Imagen-evolved vision boards
watchOS for passive tracking

App screenshots

Home

History

Plan

Scan

Built with Google Cloud, Gemini Live API, Veo 2, Imagen, and Google ADK for the Gemini Live Agent Challenge 2026.

Another take on streaming live audio from a phone: near real-time transcription with faster-whisper on my own server.

Rena’s need to remember users across sessions is where VaultMem started — an encrypted memory library for AI agents.

AI History Blog Generator

2026-03-14T00:00:00+00:00

I was curious how AI had evolved over the years — not just the big headline moments but the smaller steps in between that actually moved things forward. I wanted something I could browse casually, not a research paper or a Wikipedia rabbit hole. Just a clean walkthrough: milestone by milestone, a bit of context, who was involved, when it happened.

I couldn’t find exactly that, so I started building it. Around the same time I was also exploring what local models could actually do — running Ollama, trying different models, seeing how far you could get without sending everything to an API. That combination clicked: use a local model to do the writing, keep costs low, and let it run as long as needed. And somewhere while putting that together I thought — wait, this doesn’t have to be about AI. It could be about anything with a history. That’s how AI History Blog Generator came about — an autonomous agent that turns any historical topic into a fully-functioning Jekyll blog, one milestone at a time.

Live examples:

Default (AI history): aag1091-alt.github.io/AI-History-Blog
Custom topic (Golden Gate Bridge history): aag1091-alt.github.io/Golden-Gate-History

What It Does

Give it a topic — “the history of jazz music”, “the history of the Golden Gate Bridge”, anything — and it will:

Ask Claude to define eras and categories for the topic (only once, on first run)
Pick the next interesting milestone not yet covered
Research it via Wikipedia and arXiv
Write a full blog post and self-critique it
Fetch a relevant Wikimedia image (with NSFW filtering)
Generate a person profile page for everyone mentioned in the post
Commit and push everything to GitHub Pages
Repeat — indefinitely if you want

It tracks everything in a state.json file so you can stop it mid-run, restart later, and it picks up exactly where it left off.

How the Pipeline Works

Each post goes through a four-stage LangGraph pipeline:

Diagram: The four-stage LangGraph pipeline behind each generated post (see it in the post)
Claude or OpenAI picks the next milestone to cover. The research node looks it up on Wikipedia, on arXiv for science and tech topics, and fetches a Wikimedia image filtered for relevance and NSFW content. The write node runs a local LLM through Ollama: a WriterAgent with tool calling, using search_wikipedia, search_arxiv and write_post, runs multi-turn until write_post is called. The critique node, also on Ollama with structured output via Pydantic, checks that a year is mentioned, people are named, the section format and word count are right and there are no hallucinated URLs, and returns approved plus a list of issues. If approved, the post is published and pushed. If issues are found, a rewrite node fixes them and loops back, at most twice.
A cloud model only picks the milestone; local models write and critique each post, and a draft with issues gets up to two rewrites.

Milestone selection uses Claude (or OpenAI as fallback) via API — this is the “thinking” layer that picks what to write about. Post writing is entirely local via Ollama, so you’re not paying per token for every paragraph.

Before any of this starts, the agent checks two things to avoid covering the same ground twice: an exact slug match against state.json, and a semantic check — if the proposed milestone shares >40% keyword overlap with something already written in the same year, it’s rejected and a different one is picked.

Models Used

Task	Model
Milestone selection	Claude (Anthropic API) → OpenAI → Claude CLI
Post writing	Ollama — `qwen2.5:14b` (configurable)
Critique & structured validation	Ollama with Pydantic output

The writing model is fully swappable via .env. The critique node uses Pydantic models to enforce structured JSON output so there’s no fragile text parsing — if the model doesn’t return valid JSON, it retries.

Here’s how the models I tested compare on RAM footprint and post quality:

Model	RAM	RAM usage	Post quality
`llama3.2:3b`	~2 GB	▓▓░░░░░░░░	▓▓▓░░░░░░░
`mistral:7b`	~4 GB	▓▓▓░░░░░░░	▓▓▓▓▓░░░░░
`qwen2.5:7b`	~5 GB	▓▓▓▓░░░░░░	▓▓▓▓▓▓░░░░
`qwen2.5:14b` ✦	~9 GB	▓▓▓▓▓░░░░░	▓▓▓▓▓▓▓▓░░
`llama3.3:70b`	~43 GB	▓▓▓▓▓▓▓▓▓░	▓▓▓▓▓▓▓▓▓░

✦ default

qwen2.5:14b is the sweet spot for most machines — the quality jump from 7b is noticeable in post structure and factual coherence, and 9 GB fits comfortably on a laptop with 16 GB RAM. llama3.2:3b works if you’re just testing the pipeline; llama3.3:70b produces the best output but needs a well-specced machine or a dedicated GPU box.

Try It Yourself

You’ll need Ollama running locally and an Anthropic API key. The model is local so the only API cost is milestone selection (one call per post).

# 1. Clone and install
git clone https://github.com/aag1091-alt/AI-History-Blog-Generator.git
cd AI-History-Blog-Generator
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Configure
cp .env.example .env
# Set ANTHROPIC_API_KEY and OLLAMA_MODEL in .env

# 3. Create a blog for any topic
python -m generate --new \
  --topic "the history of jazz music" \
  --dir ~/projects/jazz-blog \
  --github-url https://github.com/you/jazz-blog

# 4. Push to GitHub and enable GitHub Pages in repo settings
cd ~/projects/jazz-blog && git push -u origin main

# 5. Generate posts
python -m generate          # one post
python -m generate --daemon # run forever (Ctrl+C to stop)
python -m generate --status # check progress

The --new command clones a Jekyll template (layouts, CSS, dark/light theme, era pages, person profiles, live search) from the template branch and wires it up for your topic. From there, every python -m generate run adds a new post and pushes it live.

What Surprised Me

The self-critique loop was the most interesting part to build. Early versions would occasionally write posts with no dates, or refer to people without naming them, or include made-up Wikipedia URLs. The critique node catches all of that with Pydantic-enforced checks and sends the post back for a rewrite with a specific list of issues. Most posts pass on the first try; the rewrite loop rarely triggers more than once.

The other thing that exceeded my expectations was the image pipeline. Wikimedia Commons has an enormous archive, and the relevance filtering (matching image title keywords against the milestone title) produces surprisingly good results. The NSFW filter — checking filenames, metadata restrictions, and category labels — has never let anything through in my testing.

What started as wanting a casual way to browse AI history turned into something I keep finding new uses for. If you have a topic you’ve always wanted a proper reference for, this is a pretty fast way to build one.

Related: Building Your Website with Jekyll and GitHub Pages walks through the Jekyll + GitHub Pages setup the generated blogs are published with, and /uses lists the home server I run local models on with Ollama.