A voice-first AI agent that connects to my Hermes agent, reads my screen on demand, and writes to my Obsidian vault.
The voice-first interface — no keyboard for core interactions.
TL;DR — A SCREENER CAN STOP HERE
3PO is a voice-first macOS companion with a British-butler persona, built on OpenAI's Realtime API. Seven tools — notes to my Obsidian vault, web search, screen vision, Gmail search/draft/send, Drive search — with permissions designed so the dangerous things are structurally impossible. A working local prototype: real voice conversations, real tools, on my Mac.
I tried talking to my main agent (Hermes) through a Discord voice channel and the latency was killing the interaction model. I also had a recurring annoyance — sending screenshots to different chat models whenever something on my screen confused me — that turned every quick question into a multi-step process. I wanted a voice-first agent with screen vision and constrained permissions, sitting between me and my main agent.
✦ NORTH STAR
“What should be a model, and what should be code?”
02 — SOLUTION
Seven tools, and permissions as product design
Seven tools live directly in the Python server — notes, web search, screen vision, Gmail search/draft/send, Drive search. Gmail's OAuth scopes make deletion structurally impossible, and sending is two-step: the model drafts, a separate explicit instruction sends. On startup 3PO reads three vault files and knows my projects before I say a word.
The vault it wakes up inside — job reports, project boards, research notes. Three of these files ride along on every session.
03 — UNDER THE HOOD
The same screens, flipped over
The hardest problem was screen capture — Realtime has no vision, so describe_screen is a dual-model trick: a screencapture subprocess, a separate synchronous vision call, and the description injected back as a tool result. macOS ties Screen Recording permission to the responsible process, which meant building a proper .app bundle just to be allowed to look.
result = subprocess.run(["screencapture", "-x", "-t", "jpg", tmp_path],
capture_output=True, timeout=5)
# ...then a SEPARATE model call, because Realtime has no vision:
payload = json.dumps({"model": "gpt-4o-mini", "messages": [{"role": "user",
"content": [{"type": "text", "text": "Describe what is on the screen concisely..."},
{"type": "image_url", "image_url":
{"url": f"data:image/jpeg;base64,{b64_image}", "detail": "low"}}]}]})
Two models, one glance — Realtime hears, a separate vision call sees, the answer comes back as speech.
✕ REJECTEDServer-VAD streaming interruptSending mic audio to OpenAI during AI speech created echo feedback — it interrupted and listened to itself. A local RMS gate + response.cancel won.
✕ REJECTEDFull Gmail scopeDeletion made structurally impossible via scopes; two-step send keeps a human in the loop.
✕ REJECTEDScreen awareness ON by defaultPrivacy-first: it stays OFF until deliberately toggled, every session.
✕ REJECTEDThe gold/black themeFully built, then reversed — replaced with the cooler Slate & Ice palette and a hand-processed ink avatar.
A NOTE ON HONESTY
“A working local prototype — single-user, not deployed. No echo cancellation on the mic capture, so barge-in over speakers is a tuning problem, not a solved one.”
05 — REFLECTION
What the build taught me
Failure modes of AI tools are architectural, not just probabilistic — the model needs structural evidence a tool works before it calls it reliably. Permission design is product design: scopes are a hard technical guarantee, not a policy promise. And latency shapes the whole interaction model — a 2–4 second round-trip feels fine in chat and completely wrong in voice.
Caleb Kim 
