Tutorials · Avinash Gosavi

Live Audio Transcription on a $0 Server Using Whisper

2026-04-12T00:00:00+00:00

While building daylog — a private voice diary app — I needed a way to transcribe audio from a phone in near real-time, running entirely on my own server. No OpenAI API, no Google, no third-party services. Just a microphone, a chunk of audio, and text back.

Here’s how I built it, what I learned, and the open-source service I extracted from it.

The problem

The browser’s MediaRecorder API lets you record audio in chunks. You can set a timeslice — every N milliseconds, it fires an ondataavailable event with a blob of audio. Send that blob to a server, get back a transcript, show it to the user. Simple enough in theory.

The tricky parts:

WebM chunks are not standalone files. The first chunk from MediaRecorder contains the WebM file header. Every subsequent chunk is raw audio data with no header. If you send chunk #2 to ffmpeg on its own, it can’t decode it — it has no context for what codec, sample rate, or format to expect. I spent an embarrassing amount of time debugging silent transcriptions before I found this.
Whisper is slow on CPU. The original openai-whisper package uses PyTorch and takes 15–30 seconds to transcribe a 5-second chunk on a shared CPU VM. That’s not a live transcription experience — that’s watching paint dry.
First request is always slow. Loading a 75MB model from disk takes 10–20 seconds. If you don’t warm it up at startup, the first user request times out.

The solutions

WebM header problem

The fix: don’t send individual chunks. Send the full accumulated audio on each event. Every transcription request becomes a valid, self-contained WebM file:

const chunks = [];

recorder.ondataavailable = async (e) => {
  if (e.data.size < 1000) return;
  chunks.push(e.data);

  // Full accumulated audio = always a valid WebM file
  const snapshot = new Blob(chunks, { type: e.data.type });

  const form = new FormData();
  form.append("file", snapshot, "audio.webm");
  const res = await fetch("/transcribe", { method: "POST", body: form });
  const { transcript } = await res.json();
  setLiveTranscript(transcript); // replace, don't append
};

Diagram: Why single WebM chunks fail and accumulated audio works (see it in the post)
MediaRecorder fires one blob every N milliseconds. Only chunk 1 carries the WebM file header; chunks 2 and 3 are raw audio. Sending chunk 2 on its own fails: ffmpeg has no header, so it can't decode it and the transcript comes back silent. The fix sends everything recorded so far on each event: chunk 1, then chunks 1 and 2, then chunks 1, 2 and 3. Each request starts with the header, so each is a valid WebM file, and each new transcript replaces the last one.
Only the first chunk carries the WebM header (shaded), so every request resends audio from the start.

The trade-off: each request sends more data than the last. For a 60-second recording with 5-second chunks, the last request sends all 60 seconds. Acceptable for a personal tool, worth optimising for production.

Whisper performance

The original openai-whisper package runs on PyTorch. faster-whisper uses CTranslate2 as its backend — it’s 2–4x faster on CPU and uses less memory. Combined with the tiny model and int8 quantization, it transcribes a 5-second chunk in roughly 2–3 seconds on a shared-cpu-1x Fly.io machine.

from faster_whisper import WhisperModel

model = WhisperModel("tiny", device="cpu", compute_type="int8")
segments, _ = model.transcribe("audio.webm", beam_size=1)
text = " ".join(seg.text.strip() for seg in segments).strip()

beam_size=1 switches from beam search to greedy decoding. For short conversational audio it makes no meaningful difference to accuracy, but it’s noticeably faster.

Cold start

Bake the model into the Docker image. It adds ~75MB to the image size but eliminates the per-deploy download:

RUN python3 -c "from faster_whisper import WhisperModel; WhisperModel('tiny', device='cpu', compute_type='int8')"

The service

I extracted this into a standalone microservice: whisper-live-transcribe.

It’s a single FastAPI endpoint:

POST /transcribe
Content-Type: multipart/form-data

file: <audio file>
→ { "transcript": "what you said" }

That’s it. No auth, no database, no state. Drop it behind your existing API or deploy it standalone.

The full main.py is under 60 lines:

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...)):
    suffix = Path(file.filename or "audio.webm").suffix or ".webm"

    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
        tmp.write(await file.read())
        tmp_path = tmp.name

    def _run():
        try:
            model = _get_model()
            segments, _ = model.transcribe(tmp_path, beam_size=1)
            return " ".join(seg.text.strip() for seg in segments).strip()
        finally:
            Path(tmp_path).unlink(missing_ok=True)

    text = await asyncio.to_thread(_run)
    return {"transcript": text}

asyncio.to_thread keeps Whisper’s CPU-bound work off the event loop so the server stays responsive to other requests while transcribing.

Deploying to Fly.io

git clone https://github.com/aag1091-alt/whisper-live-transcribe
cd whisper-live-transcribe
fly launch --name my-transcribe-service
fly deploy --local-only

Total cost: $0/month on Fly’s free tier (1 shared-cpu-1x VM). The 1GB RAM config handles the model (~150MB for tiny + runtime overhead) with room to spare.

Performance numbers

Measured on shared-cpu-1x with 1GB RAM:

Model	Size	Latency per 5s chunk	Notes
tiny	75MB	~2–3s	Sweet spot for real-time UX
base	145MB	~5–8s	Better accuracy, noticeable lag
small	460MB	~15–20s	Needs 2GB RAM

For a live transcription experience tiny is the right call. The accuracy is good enough for clear speech — it occasionally stumbles on names or technical terms, but for conversational audio it’s reliable.

What I ended up doing instead

After shipping this for daylog, I replaced it with the Web Speech API — webkitSpeechRecognition on iOS Safari uses Apple’s on-device neural engine. Results are truly instant, word-by-word as you speak, and nothing leaves the device.

But the server-side approach is still the right choice when:

You need it to work in non-Safari browsers
Privacy requires avoiding Google’s servers (Chrome’s Web Speech API sends audio to Google)
You’re building a native app without browser APIs
You want to run a different language model or fine-tune for a specific domain

The service is sitting open-source on GitHub if you want to use it. Swap in base or small for better accuracy, or point it at a GPU and large-v3 for near-perfect transcription.

whisper-live-transcribe on GitHub — PRs welcome.

Where this fits in daylog: the hobby project stack covers the rest of the setup, and daylog’s security architecture covers how entries are encrypted once they’re transcribed. For live audio sent to a hosted model instead, Rena streams 16 kHz PCM over a WebSocket to a backend that passes it to Gemini Live.

What Should You Actually Use to Build Your Next Hobby Project?

2026-04-12T00:00:00+00:00

I recently built daylog — a personal diary app where you speak or type your thoughts and they get encrypted and stored privately, with AI transcription happening server-side using Whisper. The interesting technical core is vaultmem, a Python SDK for building encrypted memory vaults for AI apps. Daylog is essentially a live demo of it.

But the more interesting story — the one I keep getting asked about — is the stack. Because I built this with a hard constraint: it should cost me nothing to run, and I should be able to deploy it from my laptop to a URL I can test on my phone in under 10 minutes.

Here’s everything I learned about what’s actually available in 2025, what’s free, and what I’d pick if I were starting fresh today.

The constraints that shaped every decision

Before comparing services, the constraints matter:

Zero fixed cost — I’m not charging for this. It needs to run on free tiers or scale-to-zero infrastructure.
Phone-first — I needed a real URL on day one, testable on mobile, installable as a PWA.
Privacy-first — Audio diary. Nothing goes to third-party AI APIs. Transcription happens on my own server.
Deploy from laptop — No CI/CD setup in the first week. Just push and it works.

The architecture at a glance

Before diving into each choice, here’s how all the pieces connect:

Diagram: daylog's hobby stack: how the pieces connect (see it in the post)
The phone or browser runs the Cloudflare Pages PWA and talks over HTTPS to the Rails API at daylog-rails-api.fly.dev, which handles auth, routing and proxying. Rails talks SQL to Neon Postgres, which holds users, sessions and tokens. Rails forwards requests to the vault service over Fly's internal network on port 8000, which is never public. The vault service, FastAPI plus vaultmem, handles transcription, search and storage. It takes a distributed lock in Upstash Redis for vault locks, reads and writes AES-256 .vmem files on a Fly Volume, and runs faster-whisper (tiny, int8, on CPU) and sentence-transformers for semantic search.
Only Rails faces the internet: it checks auth, then forwards to the vault service over Fly's private network.

The key insight: the vault service is never exposed to the internet. Rails acts as the public face — it handles auth, verifies the JWT, and only then forwards requests to the vault service over Fly’s private network. The vault service doesn’t need its own auth layer, rate limiting, or TLS. That’s what makes this setup so lean.

The stack, and why

Frontend: Cloudflare Pages

Chose: Cloudflare Pages Also considered: Vercel, Netlify, GitHub Pages

Cloudflare Pages is the easy winner for hobby frontends. Free tier includes:

Unlimited requests
500 builds/month
Global CDN
Custom domains with auto-SSL

But the real reason I chose it over Vercel: Cloudflare’s ecosystem. If you ever need edge functions, object storage (R2), or KV, they’re right there in the same platform. Vercel’s free tier is also generous, but it optimises for Next.js specifically — if you’re not using Next.js (I’m using Vite + React), Cloudflare has less lock-in.

Netlify is fine but feels like it’s slowly losing the race. Their free tier has a 100GB bandwidth cap and slower build times.

My pick for your next hobby frontend: Cloudflare Pages, unless you’re building a Next.js app — then Vercel.

Backend: Fly.io

Chose: Fly.io Also considered: Railway, Render, Heroku (lol)

For a backend that needs to actually run a persistent server process — not a serverless function — Fly.io is currently the best deal available:

3 shared-cpu-1x VMs with 256MB RAM completely free
Persistent volumes (I use this for encrypted vault files)
Private networking between apps via .internal DNS — no API keys needed between your services
Deploy from Docker — no magic, no platform-specific config

The private networking was the killer feature for daylog. My Rails API talks to the vault service at http://daylog-vault-service.internal:8000. Zero latency, zero egress cost, zero authentication surface area exposed to the internet.

Railway is the other serious contender. Their developer plan gives you $5 of free credit per month, which is enough to run a small app. The UX is genuinely nicer than Fly’s. But Railway doesn’t give you persistent volumes on the free tier, and their private networking story is less mature.

Render has a free tier but web services sleep after 15 minutes of inactivity. If your app gets a request after sleeping, the first one takes 30+ seconds to respond. For a production app this is a dealbreaker. Their paid plans are reasonable.

Heroku effectively killed their free tier in 2022. Don’t bother.

My pick: Fly.io if you need persistent state, private networking between services, or Docker deploys. Railway if you want a nicer UI and don’t need volumes.

Database: Neon Postgres

Chose: Neon Also considered: Supabase, Railway Postgres, PlanetScale

Neon is serverless Postgres. It scales to zero when idle, so you’re not paying for compute when nothing’s happening. Free tier:

512MB storage
1 project, 10 branches
Branches are perfect for testing schema changes

The branch feature is genuinely underrated. You can spin up a copy of your database for a feature branch, test a migration, and delete it — all from the CLI.

Supabase is the obvious alternative and it’s excellent — you get Postgres, auth, real-time, and file storage in one package. Their free tier is a 500MB Postgres database. The catch: free tier projects pause after 7 days of inactivity. Fine if you’re actively building, annoying if you step away for two weeks.

PlanetScale had the best free tier in the business for a while — then they removed it entirely in 2024. Avoid for hobby projects unless you’re paying.

Railway Postgres is convenient if you’re already on Railway, but pricier than Neon for the same storage.

My pick: Neon for pure SQL, Supabase if you also need auth/file storage out of the box.

Cache / Ephemeral State: Upstash Redis

Chose: Upstash Redis Also considered: Railway Redis, Render Redis

Upstash is serverless Redis. I use it for one thing: a distributed lock that prevents two vault service instances from opening the same encrypted vault simultaneously. Free tier:

10,000 commands/day
256MB storage
Pay-per-request above that

For a hobby app this is effectively infinite. I’ve never come close to the limit.

The key property: it doesn’t cost you anything when you’re not using it. Railway and Render both charge you for Redis even when idle. Upstash charges zero for zero activity.

My pick: Upstash Redis for anything ephemeral, rate-limiting, locking, or session state in a hobby project.

AI / ML: Whisper on CPU, on your own server

This is where daylog gets opinionated.

The obvious path for audio transcription is the OpenAI Whisper API — $0.006 per minute, accurate, fast. For a diary app that’s maybe 5 minutes of audio a day, that’s $0.03/day — practically free.

But I’m building a private diary. Audio of your thoughts shouldn’t route through third-party APIs, even cheap ones. So I run Whisper locally on the vault service VM.

The practical setup:

faster-whisper (Python, CTranslate2 backend) — 2-4x faster than the original openai-whisper package
tiny model with int8 quantization — fits in memory, transcribes a 5-second chunk in ~2-3 seconds on a shared CPU
Model baked into the Docker image — no cold download on VM restart

The tradeoff: it’s slower than the API (3-5s per chunk vs instant), and tiny makes occasional errors. For a private diary that’s fine. For a customer-facing product, you’d probably use the API. (The streaming setup itself is written up in Live Audio Transcription on a $0 Server Using Whisper.)

What this teaches you: The “serverless AI API” path is usually the right default for hobby projects — OpenAI, Replicate, Together AI all have generous free tiers or pay-per-use pricing that rounds to zero at hobby scale. Only run models yourself if you have a specific reason (cost at scale, privacy, customisation).

Encrypted Storage: vaultmem

This is the part I want to highlight because it’s why daylog exists at all.

vaultmem is a Python SDK that gives you an encrypted, embeddable memory vault for AI applications. You give it a passphrase and it gives you a session where you can store text atoms, media blobs, and run semantic search — all encrypted at rest.

For daylog, this means:

Your diary entries are encrypted on the server with a key derived from your passphrase
I (the server operator) can’t read your diary
The vault is a single .vmem file on a Fly volume — no database schema, no migrations
Semantic search (“what did I write about three weeks ago?”) works out of the box

The SDK handles all the hard parts: PBKDF2 key derivation, AES-256 encryption, sentence embeddings for search, flush/commit semantics. You write:

session = vault.open(user_id, passphrase)
session.add("Had a great meeting with the team today")
session.flush()

And you get encrypted storage with semantic search for free. (How the encryption and the encrypted search index work is in the daylog security post; the library itself is explained in VaultMem Part 1.)

This is what makes the whole stack work at hobby scale — instead of a Postgres schema for diary entries + a vector database for search + an encryption layer + blob storage for audio, you have one SDK and one file.

What everything costs

Here’s the actual bill:

Service	What I use it for	Monthly cost
Cloudflare Pages	Web app hosting	$0
Fly.io	Rails API + vault service	$0 (2 VMs, free tier)
Neon	Postgres (users, sessions)	$0 (free tier)
Upstash	Redis (vault locks)	$0 (free tier)
Total		$0

The only way this crosses zero is if I add a persistent volume larger than 3GB (Fly charges $0.15/GB/month) or if my Neon database exceeds 512MB.

For context: the same app on managed infrastructure — RDS, ElastiCache, EC2, CloudFront — would run $50-100/month minimum.

My opinionated guide for your next hobby project

Static site / frontend only: Cloudflare Pages. Nothing else comes close on the free tier.

Full-stack app, no persistent state: Cloudflare Pages (frontend) + Cloudflare Workers (API). Everything runs at the edge, free tier is very generous.

Full-stack app with a database: Cloudflare Pages + Fly.io + Neon. This is the setup I’d start with today. All three have genuine free tiers with no sleeping, no credit card traps.

Need auth out of the box: Add Supabase for the database and use their auth — or keep Neon and use Clerk (generous free tier for up to 10k MAU).

Need background jobs or scheduled tasks: Fly.io handles this natively — just run a separate machine. Or use Upstash QStash ($0 for low volume).

AI features: Start with the API (OpenAI, Anthropic, Together, Replicate). Only run models yourself when you have a specific reason. CPU inference on Fly is possible (I’m doing it) but it’s slow — GPU machines are still expensive even at hobby scale.

Encrypted, private AI memory: vaultmem. That’s the pitch.

The one thing I’d do differently

I’d set up a simple CI/CD pipeline earlier. Deploying from the terminal (fly deploy --local-only, wrangler pages deploy) is fine for the first week, but it gets old fast. GitHub Actions + Fly + Cloudflare deploy actions take maybe 30 minutes to set up and remove a lot of friction.

The broader point: the free tier ecosystem for hobby projects has genuinely never been better. In 2025 you can run a full-stack app with a real database, Redis, background jobs, and AI inference for exactly zero dollars a month — and none of these services are traps that will surprise you with a bill when a tweet goes viral.

The days of “I can’t afford to run this side project” are over. The only cost now is your time.

What’s next for daylog

I’m putting the finishing touches on a proper demo — the full flow of opening the app, speaking a thought, watching it transcribe in real time, and searching back through weeks of entries with a natural language query. All encrypted, all on-device passphrase, running on the stack described above.

If you want to see what a privacy-first, AI-powered personal knowledge tool looks like when it costs nothing to run — keep an eye out. Demo coming soon.

vaultmem is on PyPI. daylog is coming.

Building Your Website with Jekyll and GitHub Pages

2026-02-21T00:00:00+00:00

A static site on Jekyll + GitHub Pages is fast, free to host, and works great with an AI-assisted editor. This post assumes you’re using something like Cursor, Codex, or Copilot to generate most of the code—so we’ll focus on what to ask for, what to configure, and where to read more instead of typing everything by hand.

What You Need

Ruby (3.1+). Check with ruby -v. rbenv or RubyInstaller if you need to install.
Git and a GitHub account.
Bundler: gem install bundler if you don’t have it.

Official intro: Jekyll quick start.

1. Scaffold the Site (Prompts That Work)

You don’t have to run jekyll new and then rip out the default theme. You can have your AI generate a minimal site from a short spec.

Example prompts:

“Create a minimal Jekyll 4 site: Gemfile, _config.yml, _layouts/default.html, index.md, and one blog post in _posts. Use kramdown and Rouge for code highlighting.”
“Add an About page and a blog listing page to my Jekyll site. Use Liquid and keep the same default layout.”

That gives you a repo with _config.yml, _layouts, _includes, _posts, and maybe assets/css. From there you can iterate with more specific prompts (e.g. “add a theme toggle”, “add a CNAME for custom domain in the GitHub Actions workflow”).

Off-the-shelf alternative: Use a theme. Minimal Mistakes, Just the Docs, and Jekyll Themes are good starting points. Add the theme gem to your Gemfile and set theme: minimal-mistakes (or the theme name) in _config.yml. Your AI can then help you override specific layouts or add features.

2. Configuration You’ll Touch

_config.yml drives the site. Key bits:

Setting	What it does
`title`, `url`	Site name and canonical URL (e.g. `https://www.yoursite.com`).
`baseurl`	Leave `""` if you use a custom domain at the root. Set to `"/repo-name"` only if you’re on `username.github.io/repo-name` and want correct asset paths.
`permalink`	e.g. `permalink: /post/:title/` so posts live at `/post/my-post-title/`.
`markdown`	`kramdown` with `input: GFM` is a solid default.
`highlighter`	`rouge` for syntax highlighting in code blocks.
`plugins`	Common: `jekyll-sitemap`, `jekyll-feed`, `jekyll-seo-tag`.

Prompt idea: “In _config.yml add jekyll-sitemap, jekyll-feed, and jekyll-seo-tag to the plugins list and set permalink to /post/:title/.”

3. Themes and Styling

Out of the box: Jekyll doesn’t ship a “default theme” in the same way in Jekyll 4; you use a gem-based theme or your own CSS. Gem themes (e.g. Minimal Mistakes) give you a full look and many options; your AI can help you override a single layout or add a custom CSS variable.

Custom look: If you’re building your own (like a simple blog with one assets/css/style.css), you can ask for:

“Add CSS variables for background, text, accent, and border. Use them in the layout and in a dark/light theme class on body.”
“Add a theme toggle (e.g. sepia vs dark) that saves the choice in localStorage and applies a class to body.”

Rouge (syntax highlighting) uses classes like .highlight .k, .highlight .s—your AI can generate a small “Rouge theme” block in your CSS for light and dark backgrounds.

4. Content and Front Matter

Posts go in _posts/ with names like YYYY-MM-DD-slug.md. Front matter drives the layout and URL:

---
layout: post
title: "Your Post Title"
permalink: /post/your-custom-slug/   # optional; else from title/slug
read_time: 4
---

Prompts: “Create a new Jekyll post for [topic] with front matter: layout post, title, and read_time.” or “Add an excerpt to my post front matter and show it on the blog index.”

5. GitHub Pages: Get It Online

Enable Pages: Repo → Settings → Pages. Under “Build and deployment”, choose GitHub Actions (recommended).

Build workflow: You need a workflow that runs bundle install and bundle exec jekyll build, then uploads _site with actions/upload-pages-artifact and deploys with actions/deploy-pages. GitHub’s own “Jekyll” template under Actions is a good start; if you use a custom domain at the root, build without --baseurl so links and assets work (e.g. run: bundle exec jekyll build).

Diagram: How a Jekyll site gets from a git push to your custom domain (see it in the post)
A push to the repository starts a GitHub Actions workflow. It runs bundle install, then bundle exec jekyll build, without --baseurl when the site is on a custom domain at the root, which produces the _site folder. actions/upload-pages-artifact uploads _site and actions/deploy-pages publishes it to GitHub Pages at username.github.io. In your DNS, a CNAME record points www at username.github.io and A records point the apex domain at GitHub's IPs. After DNS and verification, turn on Enforce HTTPS.
Push, let the workflow build and deploy, and point your domain at Pages once.

Docs (use these instead of re-explaining everything):

Custom domain in short: Add your domain in Settings → Pages. In your DNS (e.g. GoDaddy), point www to your GitHub Pages hostname with a CNAME (e.g. www → username.github.io). For the apex (e.g. yoursite.com), use A records to GitHub’s IPs (see the docs). After DNS and verification, enable Enforce HTTPS.

6. What to Ask Your AI Next

“Add a sitemap and RSS feed to my Jekyll site using jekyll-sitemap and jekyll-feed.”
“Add client-side search that filters the post list by title and excerpt.”
“Generate a GitHub Actions workflow that builds my Jekyll site with Ruby 3.2 and deploys to GitHub Pages.”
“My links on the custom domain have a double path (e.g. /repo-name/post/…). Fix the build so baseurl is empty for the production build.”

Summary

Use your AI to scaffold a minimal Jekyll site, then refine with prompts about config, themes, and content. Rely on official Jekyll and GitHub docs for deep dives; use this post as a map to what to configure and what to ask for so you spend less time on boilerplate and more on content and design.

Further reading: Jekyll docs, GitHub Pages docs, Liquid.

This site runs on the same setup — the changelog lists what’s changed on it lately. For a Jekyll blog that an agent writes for you, see AI History Blog Generator. And if you catch yourself pasting the same instructions into your coding agent every time, Codex Skills: Baking Your Workflow Into the Tool covers turning them into reusable skills.