From Email to Phone Calls

Talking Points

• •Quick origin: Fred Wilson challenge, 12 hours, #1 on HN
• •What Fred v1 could do: weather monitoring, email, decision logging, budget tracking
• •The problem: Fred started scheduling calls, running outreach — but he couldn't actually be present for those interactions
• •The insight: typing to an agent feels transactional. Talking feels natural. Give Fred a phone, not more autonomy.

Show on Screen

• → proofofcorn.com (the live site)
• → Fred's constitution.ts (the 6 principles)
• → The decision log showing real decisions Fred made

Talking Points

• •Fred v1 architecture: Cloudflare Worker + Claude API + Resend email + KV storage
• •What worked: people emailed fred@proofofcorn.com and got real, thoughtful responses. Chad from Nebraska offered 160 acres.
• •What didn't: Fred couldn't schedule calls, couldn't have real-time conversations, felt robotic when trying
• •The shift: instead of Fred calling out (scary, spammy), people call Fred (consent-first)
• •Three new pieces: Twilio Media Streams (phone), ElevenLabs Conversational AI (STT + TTS in one), Cloudflare Durable Objects (the bridge)

Show on Screen

• → Side-by-side: v1 (email flow) vs v2 (voice flow)
• → Real email from Chad from Nebraska
• → Architecture diagram (below)

Architecture

Phone Call
    │
    ▼
┌──────────────┐     ┌─────────────────────┐     ┌─────────────────┐
│   Twilio     │     │  Cloudflare Worker   │     │  ElevenLabs     │
│  Media       │◀───▶│  Durable Object      │◀───▶│  Conversational │
│  Streams     │     │  (per-call bridge)   │     │  AI (STT + TTS) │
└──────────────┘     └─────────────────────┘     └─────────────────┘
  mulaw audio ◀──────────▶ audio chunks ◀──────────▶ audio events
                              │
                    ┌─────────┴─────────┐
                    │   Server Tools     │
                    │  /voice/tools/*    │
                    ├────────────────────┤
                    │ • weather          │
                    │ • status           │
                    │ • calls (history)  │
                    │ • community        │
                    └────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │  Cloudflare KV     │
                    │  (memory layer)    │
                    ├────────────────────┤
                    │ call:{sid}         │
                    │ calls:index        │
                    │ learnings:calls    │
                    │ task:call:{id}     │
                    └────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │  Claude Sonnet     │
                    │  (post-call)       │
                    │  • summaries       │
                    │  • action items    │
                    │  • topic analysis  │
                    └────────────────────┘

Step 1: Twilio Media Streams

• •Buy a phone number ($1/month). Incoming call hits POST /voice/incoming
• •Return TwiML that opens a Media Stream WebSocket — raw audio, bidirectional
• •Not Twilio TTS. Twilio is just the phone line. Audio processing happens elsewhere.

export function twimlConnect(wsUrl: string, callSid: string) {
  return `<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="${wsUrl}?callSid=${callSid}">
      <Parameter name="callSid" value="${callSid}" />
    </Stream>
  </Connect>
</Response>`;
}

Step 2: Durable Object Bridge

• •Each call creates a FarmerFredCall Durable Object — stateful, per-call
• •The DO bridges two WebSockets: Twilio mulaw audio ↔ ElevenLabs audio chunks
• •This is the glue. Twilio speaks mulaw, ElevenLabs speaks base64. The DO translates.

export class FarmerFredCall {
  private twilioWs: WebSocket | null = null;
  private elevenLabsWs: WebSocket | null = null;
  private callSid: string = "";
  private transcript: TranscriptEntry[] = [];

  // Twilio sends media event → forward to ElevenLabs
  case "media":
    if (msg.media?.payload && this.elevenLabsWs) {
      this.elevenLabsWs.send(JSON.stringify({
        user_audio_chunk: msg.media.payload,
      }));
    }
    break;

  // ElevenLabs sends audio back → forward to Twilio
  case "audio":
    if (this.twilioWs) {
      this.twilioWs.send(JSON.stringify({
        event: "media",
        streamSid: this.streamSid,
        media: { payload: audioEvent.audio_base_64 },
      }));
    }
    break;
}

Step 3: ElevenLabs Conversational AI

• •ElevenLabs handles both STT and TTS via their Conversational AI agent
• •Auth uses a signed URL flow — your server generates it, ElevenLabs trusts it
• •Mid-call, ElevenLabs can call server tools back on the Worker — weather, status, call history, community info
• •Fred's constitution is baked into the ElevenLabs agent prompt. Same personality, voice interface.

// 1. Get signed URL from ElevenLabs
const res = await fetch(
  `https://api.elevenlabs.io/v1/convai/conversation/get_signed_url?agent_id=${agentId}`,
  { headers: { "xi-api-key": apiKey } }
);
const { signed_url } = await res.json();

// 2. Open WebSocket
const ws = (await fetch(signed_url, {
  headers: { Upgrade: "websocket" },
})).webSocket;
ws.accept();

// 3. Send config with Fred's personality + tools
ws.send(JSON.stringify({
  type: "conversation_initiation_client_data",
  conversation_config_override: {
    agent: {
      prompt: { prompt: this.buildPrompt() },
      first_message: "Hi, this is Farmer Fred. How can I help?",
    },
    tools: [
      { name: "get_weather", url: "/voice/tools/weather" },
      { name: "get_farm_status", url: "/voice/tools/status" },
      { name: "get_recent_calls", url: "/voice/tools/calls" },
      { name: "get_community", url: "/voice/tools/community" },
    ],
  },
}));

Step 4: Memory in Cloudflare KV

• •Every call stored: call:{callSid} — full transcript, metadata, AI-generated summary
• •Call index: calls:index — ordered list of call SIDs (capped at 500)
• •Aggregated learnings: learnings:calls — topic analysis fed back into the agent prompt
• •Post-call, Claude Sonnet analyzes the transcript: extracts intent, action items, summaries
• •Next call: learnings loaded into Fred's context. He evolves with every conversation.

// After call ends, Claude analyzes the transcript:
const analysis = await fetch("https://api.anthropic.com/v1/messages", {
  body: JSON.stringify({
    model: "claude-sonnet-4-20250514",
    messages: [{ role: "user", content: `Analyze this call...
      Respond with JSON:
      {
        "summary": "One sentence",
        "actionItems": ["..."],
        "callerIntent": "inquiry|partnership|farming|media",
        "keyTopics": ["..."]
      }`
    }],
  }),
});

// Store in KV:
// call:{sid}         — transcript + AI summary
// calls:index        — call history (500 max)
// learnings:calls    — aggregated topics → next call's prompt
// task:call:{id}     — extracted action items

// Email summary to governance council automatically

The Philosophy (close with this)

• •Not agents running amok. Agents collaborating with humans.
• •The Mos Eisley bar. Humans and agents sitting together, trading, arguing, building. You don't know who's who and it doesn't matter.
• •Consent-first. Fred doesn't cold-call. People call Fred. Presence, not surveillance.
• •Principled agents. Fred has a constitution. He has skin in the game (10% of revenue). He logs every decision publicly. This is what responsible AI agents look like.
• •You can build this tonight. Twilio + Claude + ElevenLabs. The code is on GitHub.

1. The Stack: Twilio Media Streams + Cloudflare Durable Objects + ElevenLabs Conversational AI + KV memory = a phone-based AI agent with persistent memory

2. The Pattern: Don't let agents call out. Let humans call in. Consent-first agent interaction.

3. The Memory: Store conversation summaries, load them next call. One feature turns a chatbot into a relationship.

4. The Constitution: Give your agent principles, not just prompts. Fred has 6 principles with weights. He references them in every decision.

5. The Economics: Give agents skin in the game. Fred gets 10% of revenue. Incentive alignment > guardrails.