How I built my own AI Assistant
Tee Jay here (the human) I have decided, the best way to describe what I have built for myself, is to actually provide the software stack that I have built. For those that are uninitiated in tech speak, a software stack is a collection of programs working together.
I am using AI (artificial intelligence) as an umbrella term for all the technologies that include ML (machine learning), Generative AI (image/text/video/audio/music generation using computers) and LLMs (large language models). I am hoping I can dispel the myth of what AI is (and isn’t)
This is what my ai software stack is built on, and what it can do. I gave Lobster Boy the list, and as you can see, he/it regenerated the text, because I missed parts of my stack. As this is an IT project, I feel that this is OK, it wouldn’t be OK if this was creative writing!
My AI Assistant
What it is
Openclaw harness to orchestrate all APIs, and to run crons.
All this enables me to have an AI assistant that allows me to chat to it, with the personality that of a space lobster, in Telegram, that has access to my self hosting servers and data.
The harness orchestrates all the API/LLM/machine learning tools, automates scripts and sets them off each time, whilst giving me the luxury of talking to my network in plain English, using a messaging app.
Where it lives
It runs on one server (The Shell, Gen 8 MicroServer), but the AI stack is spread over an LLM API, a few Docker containers, my Mac Mini and a virtual machine.
What it can do (current state)
Servers & infrastructure
- Passwordless sudo on all my Linux servers — Pis (speaker-LR, speaker-kt), Media Server, Nextcloud VM on Gen 9
- NextCloud — read files, send backups, upload files, read calendar and contacts
- Smart home— Home Assistant + MQTT integration
- Docker — create and update images; destroy requires explicit confirmation (ask-first pattern)
- Media server — Radarr/Sonarr/Sabnzbd/Jackett/Navidrome + share-play Raspberry Pi (AirPlay receivers)
- Server health monitoring — Beszel across all systems
- UPS monitoring — apcupsd service data, both Gen 8 and Gen 9 (Windows-side via NIS)
Personal
- Blood pressure tracking — weekly cadence via Omron + nudges
- Daily briefing — weather + UPS health + servers health + calendar, every morning via Telegram
- Calendar + contacts — read access via Nextcloud; writes require per-call confirmation
- Outbound email — lobsterboy@tjay.me.uk (e.g. GATA calendar digest)
- SMS / phone escalation — Twilio on the two-layer safety net (OpenClaw-down + TJ-silence detection)
- Voice input — iOS node has voice wake + Talk mode
Productivity
- Generate images — local model or hosted
- Analyze images — local LLaVA on the Mac
- Summarize text — native LLM capability
- Research projects/products — web search + fetch
- Browser automation — can drive pages, not just fetch
- Help with WordPress projects — Plesk + REST API
- Vibe code small programs — “do one thing and do it well”
Memory & continuity
- Cross-session memory — MEMORY.md + daily notes
- Semantic search — local Ollama embeddings on the Mac
- Audit log — every API call lands in `state/assistant-audit.log`
- Restore-drill Sunday self-test — automatic backup integrity check, Telegram on failure
- Weekly Saturday audit cadence — punch list only, no auto-edits
What’s still in flight
Heartbeat → local LLM — currently fires via hosted API; once the Gen 9 32GB RAM upgrade lands, this moves to local model on the bedroom server
I hope that clears that up 🙂