Self-Hosted Open Source Agent
VerifiedFully self-hosted agent stack using open-source models and tools for maximum control and data privacy.
Last verified: 2026-10-29
Orchestration
Memory
LangGraph checkpointer backend for conversation state.
Fast session cache and pub/sub for real-time updates.
RAG
Open-source embedding database, runs locally or in Docker.
Document loaders, text splitters, and retrieval chains.
MCP / Tools
Define tools as Python functions with type hints.
Read/write files, process documents locally.
Extract content from websites.
Custom integrations with internal systems.
Eval / Tracing
Self-hosted LLM observability and evaluation.
Standard tracing protocol for distributed systems.
Models
Open model via Ollama or vLLM for main reasoning.
MoE model for complex reasoning tasks.
Open-source embeddings for RAG.
Cost & Performance Notes
Cost
Hardware costs: ~$3-5K for capable GPU server or ~$500-1500/month cloud GPU. No per-token API costs.
Latency
Depends heavily on hardware. With A100: P50 ~1s, P95 ~3s. Consumer GPUs significantly slower.
Caveats & Considerations
- Requires significant DevOps/MLOps expertise
- Open models lag behind GPT-4/Claude on complex reasoning
- GPU infrastructure is expensive and complex
- No SLA - you're responsible for uptime
- Model updates require manual deployment
For teams requiring complete data sovereignty or operating in air-gapped environments, this self-hosted stack provides full control over the entire AI pipeline. It trades convenience for maximum flexibility and privacy.
Infrastructure Overview
┌─────────────────────────────────────────────────────┐
│ Load Balancer │
│ (nginx/Traefik) │
├─────────────────────────────────────────────────────┤
│ FastAPI Backend │
│ (Agent API + WebSocket) │
├─────────────────────────────────────────────────────┤
│ LangGraph │
│ (Agent orchestration layer) │
├───────────────────┬─────────────────────────────────┤
│ Model Server │ Vector Store │
│ ┌─────────────┐ │ ┌─────────────────────────────┐│
│ │ Ollama │ │ │ Chroma ││
│ │ or │ │ │ (embeddings + search) ││
│ │ vLLM │ │ │ ││
│ └─────────────┘ │ └─────────────────────────────┘│
│ │ │ │ │
│ GPU Server │ PostgreSQL │
│ (A100/H100) │ (state + checkpoints) │
├───────────────────┴─────────────────────────────────┤
│ Observability │
│ Phoenix │ Prometheus │ Grafana │
└─────────────────────────────────────────────────────┘Docker Compose Example
version: "3.8"
services:
ollama:
image: ollama/ollama:latest
volumes:
- ollama_data:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
chroma:
image: chromadb/chroma:latest
volumes:
- chroma_data:/chroma/chroma
postgres:
image: postgres:16
environment:
POSTGRES_DB: agent
POSTGRES_USER: agent
POSTGRES_PASSWORD: secret
agent:
build: .
depends_on:
- ollama
- chroma
- postgres
environment:
- OLLAMA_HOST=ollama:11434
- CHROMA_HOST=chroma:8000
- DATABASE_URL=postgresql://agent:secret@postgres/agentTrade-offs
| Aspect | Self-Hosted | Cloud APIs |
|---|---|---|
| Data Privacy | ✅ Complete control | ❌ Data leaves network |
| Cost (high volume) | ✅ Fixed infrastructure | ❌ Per-token pricing |
| Quality | ⚠️ Open models lag | ✅ State-of-art models |
| Maintenance | ❌ DIY everything | ✅ Managed service |
| Latency | ⚠️ Depends on hardware | ✅ Optimized infra |
When to Use This Stack
- Strict data residency or compliance requirements (HIPAA, SOC2, government)
- High-volume use cases where API costs become prohibitive
- Teams with strong ML/DevOps capabilities
- Air-gapped or on-premise deployments
Something wrong or outdated?
Help us keep this stack accurate.