Local-first · Avinash Gosavi

Live Audio Transcription on a $0 Server Using Whisper

2026-04-12T00:00:00+00:00

While building daylog — a private voice diary app — I needed a way to transcribe audio from a phone in near real-time, running entirely on my own server. No OpenAI API, no Google, no third-party services. Just a microphone, a chunk of audio, and text back.

Here’s how I built it, what I learned, and the open-source service I extracted from it.

The problem

The browser’s MediaRecorder API lets you record audio in chunks. You can set a timeslice — every N milliseconds, it fires an ondataavailable event with a blob of audio. Send that blob to a server, get back a transcript, show it to the user. Simple enough in theory.

The tricky parts:

WebM chunks are not standalone files. The first chunk from MediaRecorder contains the WebM file header. Every subsequent chunk is raw audio data with no header. If you send chunk #2 to ffmpeg on its own, it can’t decode it — it has no context for what codec, sample rate, or format to expect. I spent an embarrassing amount of time debugging silent transcriptions before I found this.
Whisper is slow on CPU. The original openai-whisper package uses PyTorch and takes 15–30 seconds to transcribe a 5-second chunk on a shared CPU VM. That’s not a live transcription experience — that’s watching paint dry.
First request is always slow. Loading a 75MB model from disk takes 10–20 seconds. If you don’t warm it up at startup, the first user request times out.

The solutions

WebM header problem

The fix: don’t send individual chunks. Send the full accumulated audio on each event. Every transcription request becomes a valid, self-contained WebM file:

const chunks = [];

recorder.ondataavailable = async (e) => {
  if (e.data.size < 1000) return;
  chunks.push(e.data);

  // Full accumulated audio = always a valid WebM file
  const snapshot = new Blob(chunks, { type: e.data.type });

  const form = new FormData();
  form.append("file", snapshot, "audio.webm");
  const res = await fetch("/transcribe", { method: "POST", body: form });
  const { transcript } = await res.json();
  setLiveTranscript(transcript); // replace, don't append
};

Diagram: Why single WebM chunks fail and accumulated audio works (see it in the post)
MediaRecorder fires one blob every N milliseconds. Only chunk 1 carries the WebM file header; chunks 2 and 3 are raw audio. Sending chunk 2 on its own fails: ffmpeg has no header, so it can't decode it and the transcript comes back silent. The fix sends everything recorded so far on each event: chunk 1, then chunks 1 and 2, then chunks 1, 2 and 3. Each request starts with the header, so each is a valid WebM file, and each new transcript replaces the last one.
Only the first chunk carries the WebM header (shaded), so every request resends audio from the start.

The trade-off: each request sends more data than the last. For a 60-second recording with 5-second chunks, the last request sends all 60 seconds. Acceptable for a personal tool, worth optimising for production.

Whisper performance

The original openai-whisper package runs on PyTorch. faster-whisper uses CTranslate2 as its backend — it’s 2–4x faster on CPU and uses less memory. Combined with the tiny model and int8 quantization, it transcribes a 5-second chunk in roughly 2–3 seconds on a shared-cpu-1x Fly.io machine.

from faster_whisper import WhisperModel

model = WhisperModel("tiny", device="cpu", compute_type="int8")
segments, _ = model.transcribe("audio.webm", beam_size=1)
text = " ".join(seg.text.strip() for seg in segments).strip()

beam_size=1 switches from beam search to greedy decoding. For short conversational audio it makes no meaningful difference to accuracy, but it’s noticeably faster.

Cold start

Bake the model into the Docker image. It adds ~75MB to the image size but eliminates the per-deploy download:

RUN python3 -c "from faster_whisper import WhisperModel; WhisperModel('tiny', device='cpu', compute_type='int8')"

The service

I extracted this into a standalone microservice: whisper-live-transcribe.

It’s a single FastAPI endpoint:

POST /transcribe
Content-Type: multipart/form-data

file: <audio file>
→ { "transcript": "what you said" }

That’s it. No auth, no database, no state. Drop it behind your existing API or deploy it standalone.

The full main.py is under 60 lines:

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...)):
    suffix = Path(file.filename or "audio.webm").suffix or ".webm"

    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
        tmp.write(await file.read())
        tmp_path = tmp.name

    def _run():
        try:
            model = _get_model()
            segments, _ = model.transcribe(tmp_path, beam_size=1)
            return " ".join(seg.text.strip() for seg in segments).strip()
        finally:
            Path(tmp_path).unlink(missing_ok=True)

    text = await asyncio.to_thread(_run)
    return {"transcript": text}

asyncio.to_thread keeps Whisper’s CPU-bound work off the event loop so the server stays responsive to other requests while transcribing.

Deploying to Fly.io

git clone https://github.com/aag1091-alt/whisper-live-transcribe
cd whisper-live-transcribe
fly launch --name my-transcribe-service
fly deploy --local-only

Total cost: $0/month on Fly’s free tier (1 shared-cpu-1x VM). The 1GB RAM config handles the model (~150MB for tiny + runtime overhead) with room to spare.

Performance numbers

Measured on shared-cpu-1x with 1GB RAM:

Model	Size	Latency per 5s chunk	Notes
tiny	75MB	~2–3s	Sweet spot for real-time UX
base	145MB	~5–8s	Better accuracy, noticeable lag
small	460MB	~15–20s	Needs 2GB RAM

For a live transcription experience tiny is the right call. The accuracy is good enough for clear speech — it occasionally stumbles on names or technical terms, but for conversational audio it’s reliable.

What I ended up doing instead

After shipping this for daylog, I replaced it with the Web Speech API — webkitSpeechRecognition on iOS Safari uses Apple’s on-device neural engine. Results are truly instant, word-by-word as you speak, and nothing leaves the device.

But the server-side approach is still the right choice when:

You need it to work in non-Safari browsers
Privacy requires avoiding Google’s servers (Chrome’s Web Speech API sends audio to Google)
You’re building a native app without browser APIs
You want to run a different language model or fine-tune for a specific domain

The service is sitting open-source on GitHub if you want to use it. Swap in base or small for better accuracy, or point it at a GPU and large-v3 for near-perfect transcription.

whisper-live-transcribe on GitHub — PRs welcome.

Where this fits in daylog: the hobby project stack covers the rest of the setup, and daylog’s security architecture covers how entries are encrypted once they’re transcribed. For live audio sent to a hosted model instead, Rena streams 16 kHz PCM over a WebSocket to a backend that passes it to Gemini Live.

How daylog Keeps Your Diary Private (Even from Itself)

2026-04-12T00:00:00+00:00

daylog is a voice diary. You open it, tap the mic, say what’s on your mind, and it saves. You can search your entries semantically — type “frustrated” and it surfaces the day you wrote “couldn’t make progress, everything felt blocked” even though that exact word never appears.

That search working at all, while keeping your diary private, requires a few things to be true simultaneously: the content must be encrypted at rest, the search index must also be encrypted, and nothing should leave the device to answer a query. This post explains how each of those holds.

How your words travel

The diagram traces a single entry from your mouth to disk, and back to your eyes.

Diagram: How a daylog entry travels to disk and back (see it in the post)
Saving goes down five layers and reading comes back up the same layers. You speak, for example "went for a run today". The browser transcribes it on-device, so nothing is sent yet. HTTPS carries it encrypted in transit by TLS. The vault embeds the entry and encrypts text and vector together with AES-256-GCM using your key; when reading, it decrypts with your key only, in memory, not logged. On disk the entry is stored as ciphertext such as 7f3a a4c8 e12d, unreadable. Reading reverses the path: the vault decrypts, HTTPS carries it back, the browser shows plain text and you see it.
Readable only at the top two layers. In transit and on disk it's ciphertext, and only the vault, with your key, turns it back into words.

Your words are readable on your device and readable back in your browser. Everything in between — transit, storage — is ciphertext. Notice the saving path says “text + vector encrypted”: the search index is encrypted too, not just the content. More on that below.

What vaultmem actually does

The vault service uses vaultmem — a library I built for giving AI agents a private, searchable long-term memory. In daylog it does three things every time you save an entry:

1. Classifies the entry automatically.

Every entry is run through a fast, rule-based classifier (no LLM involved) that assigns a memory type:

EPISODIC — a time-anchored personal experience. “Yesterday I went for a run.” Has a temporal marker + first-person reference.
PERSONA — a habitual trait or preference. “I usually start work before 8am.” Has a frequency word + first-person framing.
PROCEDURAL — a how-to or sequence. “First compile, then run tests.” Has sequential markers.
SEMANTIC — a timeless fact. Everything else.

Most diary entries classify as EPISODIC. The type determines how the memory is weighted and how it ages — episodic memories have a natural recency decay, semantic ones don’t.

2. Embeds the entry as a 384-dimensional vector.

Before the entry is encrypted, it is run through sentence-transformers/all-MiniLM-L6-v2 — a 22M parameter model running locally inside the vault container. The model converts your text into a 384-float vector that captures the semantic meaning of what you wrote. (sentence-transformers is the library that came out of the Sentence-BERT paper.)

“went for a run today” → [0.312, -0.087, 0.441, ..., 0.198] (384 values)

This vector is how search finds relevant entries later. It is stored alongside the text.

3. Encrypts text + vector together and writes to disk.

The entry object — text, classification, timestamp, and embedding vector — is serialised to JSON, compressed with zlib, and encrypted with AES-256-GCM using the master encryption key (MEK). The result is appended to a .vmem file on disk as IV + length + ciphertext + GCM tag.

The embedding is encrypted. The search index — the thing that makes semantic search work — is not sitting in a plaintext index somewhere. It is inside the same ciphertext block as your diary entry. Nobody can search your vault without your key.

How search works end to end

When you type “frustrated” into the search bar:

1. The vault service embeds your query.

“frustrated” → [0.289, -0.041, 0.509, ...] (the same 384-dim model, same vector space as your stored entries).

2. It decrypts all stored embeddings from disk.

Every .vmem entry is decrypted in RAM. The vault service now holds a matrix of shape (N entries) × 384.

3. It runs a batched dot product.

Since both vectors are unit-normalised, dot product = cosine similarity. One matrix multiply gives a relevance score for every entry simultaneously.

4. It filters, ranks, and returns.

Entries with score below 0.3 are dropped. The rest are sorted and the top 8 returned — with their decrypted text.

The decrypted entries exist only in RAM during this step. They are never written back to disk.

Why this finds meaning, not just keywords.

Two sentences with similar meaning will have similar vectors even if they share no words. “couldn’t make any progress today, everything felt blocked” scores ~0.62 against “frustrated” because the model has learned that these concepts occupy the same region of the vector space.

This is the same reason “angry” might surface an entry about a deploy failing at midnight, or “energised” might surface one about a flow state — the words you actually wrote don’t have to match the words you search for.

The memory hierarchy

vaultmem stores entries at three granularity levels:

ATOM — a single diary entry. This is what you write.
COMPOSITE — an auto-generated aggregation of related atoms. A weekly summary, or a cluster of entries about the same topic. Not yet used in daylog but the foundation is there.
AFFINITY — a detected standing pattern. If the vault notices you mention sleep problems every week, it could surface that as a persistent affinity. This is where it starts to feel less like a diary and more like a memory that knows you.

Search queries all three tiers in priority order — AFFINITY first, then COMPOSITE, then ATOM — so a standing pattern surfaces above any individual entry that mentions the same thing.

The one trade-off: AI summarise

When you tap Summarise this day, the browser’s already-decrypted entries for that day are sent to Groq (Llama 3.3 70B) via the Rails API. What Groq receives:

09:10 AM: Went for a run today.
08:30 PM: Good session, finally fixed that bug.

Time + text for one day. No name, no metadata, nothing from other days. The summary is returned and displayed — it’s never saved back into the vault.

Everything else — adding entries, browsing, searching — is vault-only. No external API. Summarise is the one feature that trades privacy for convenience, and it’s a button you tap deliberately.

If that trade-off isn’t right for you, remove GROK_API_KEY from the Rails API config and everything else works unchanged.

Data at rest

Where	What	Status
Fly.io volume (`/data/vaults/`)	`.vmem` — text + embedding vectors, all encrypted	AES-256-GCM
Postgres (Neon)	email, bcrypt password hash	Passwords hashed; DB encrypted at rest
Browser sessionStorage	session_id, derived passphrase	Cleared on tab close
Browser localStorage	auth JWT	Cleared on sign out
Vault service RAM	MEK, decrypted atoms during active session	Never written to disk

The short version

Voice input: transcribed on-device by Apple’s neural engine. Nothing leaves your phone at this step.
Saving: entry is classified, embedded, then encrypted. Text and search vector hit disk together, both as ciphertext.
Searching: vault decrypts embeddings in RAM, runs cosine similarity, returns matching plaintext. No external API.
Summarising: sends that day’s entries to Groq. Optional, explicit, one day at a time.

The encrypted search index is the part most people don’t expect. Normally if you want full-text or semantic search, you need a plaintext index somewhere — Elasticsearch, pgvector, whatever. vaultmem stores the vectors encrypted alongside the content, so you get semantic search without giving the index to anyone.

Try it yourself

The app is live. You can log in with the demo account and poke around — all entries are pre-seeded across the last two months.

daylog-ds3.pages.dev

email:    demo@daylog.app
password: demo1234

Things worth trying:

Search (bottom nav → Search tab): type frustrated, flow state, couldn't sleep, or good run — these find entries by meaning, not exact words
Summarise: navigate to any day that has entries, scroll past the list, tap Summarize this day — it reads everything you logged that day and reflects back themes and mood
Voice: tap the mic, say something, watch the live transcription appear word by word

The rest of the stack (hosting, transcription, costs) is in my hobby project stack, and the open-source Whisper transcription service I extracted from daylog has its own post. Episodic, semantic, procedural and persona memory, embeddings, semantic search and decay each get a short definition in the agent memory glossary.