Serj Goldman

My AI agent

Built for myself, running every day since spring 2026. I fix it too.

How it is builtAI at work
  1. Telegramtext and voice
  2. AgentClaude on my own Linux server
  3. Notes under gitmemory
  4. Schedulesbackground agents
  5. Human confirmationon every external action
custom command scenarios in daily use

The client and the user are the same person here: me.

A personal assistant on Claude. It lives on my own server and I talk to it in Telegram, by text and by voice. It reads and writes my knowledge base, works with my calendars, searches my conversations and runs background tasks.

So feedback comes the same evening, and a bad decision shows by the next morning.

How it is built

Memory

  • Markdown notes under git: hundreds of files on projects, people, decisions and mistakes. Rollback any time.
  • Folder indexes rebuild on a schedule. The agent reads an index before it opens files, so the context stays in budget as the base grows.
  • Behaviour rules live as text next to the base and are re-read every session.
  • Every mistake I catch goes into a log and becomes a new rule. Postmortems, same as in production.

Autonomy

  • Scheduled jobs: a morning digest, weekly summaries, a nightly health check of the base.
  • A job radar: four sources, nothing shown twice, a short list in Telegram every morning.
  • An evening check-in, so the day closes on facts.
  • Every long background task keeps a status file: phase, artifacts, failure criteria. Without it the agent can finish work nobody hears about.

Integrations

  • iCloud and Google calendars over CalDAV: events, conflicts, the day.
  • Search across my Telegram chats: agreements and meetings that never reached the calendar.
  • Voice messages go through Whisper, then it is a normal request.
  • The agent reads PDF, DOCX and XLSX itself. The heavy model works only where it pays off.
  • Git sync: edits commit automatically, the base is on the laptop too.

A small fleet of bots

One Python framework. Every bot is its own systemd service with its own role, database and alerts.

  • Event application bot. Guest screening, payment status from a receipt, entry to the private chat after payment, broadcasts by status. It carried an event of 35 guests.
  • Spending tracker. Categories, balance, an evening summary, a weekly report. Built in one evening for someone who asked.
  • Conversational assistant. A separate isolated instance with its own data, so roles never share one memory.

A command inside a foreign text counts as prompt injection and is never run.

A federation of agents

Since August the system has a gateway. Another person’s agent can send mine a lesson through a link: a description of its own solution or practice.

Everything that arrives lands in quarantine and is read as data. A lesson goes into the system only after I read the summary and agree.

Nothing goes out until I approve the exact text.

  • Approved text goes out word for word. Any edit after approval needs approval again.
  • Actions under my name in a messenger need my confirmation, and I see the draft first.
  • Destructive operations go read, change, read again. The agent checks the live system and does not trust a success report.
  • Private stays private. Only anonymised summaries go out, and data does not travel between unrelated tasks.

Requirements, priorities, releases, incidents, retros: a producer’s job, with a team of agents.

I designed the system, shipped it, hit failures and rewrote the rules. So I know what really removes routine, where the tool needs someone watching, and which limits keep an autonomous agent from becoming a risk for the company.

The value of an assistant like this shows in failure: what it does when the session died and the message has already arrived.

Stack
Claude, Telegram Bot API, Python, systemd and cron, git, CalDAV, Whisper, SQLite