Building a self-hosted open source AI stack is no longer reserved for research teams. In 2026, an ops team can run an LLM on its own servers, offer colleagues a chat interface, plug a RAG into its documentation, automate alert triage and trace every call, without any data leaving its infrastructure. This guide compares the building blocks of that self-hosted AI infrastructure, layer by layer.
The difficulty is no longer finding tools but choosing between them. Ollama, vLLM, llama.cpp, Open WebUI, LangChain, LlamaIndex, CrewAI, Dify, n8n, Langfuse, Phoenix: every project has its place, but not necessarily in your context. And every building block you add is one more service to deploy, update, secure and monitor.
This pillar guide organises these projects into five layers, compares their strengths and limits in a table, then suggests typical combinations. Each building block has a dedicated article, linked below, with a step-by-step guide and operational advice.
The five layers of a self-hosted open source AI stack
- Inference: the server that loads the model and answers requests (Ollama, vLLM, llama.cpp).
- Interface: the chat application for users (Open WebUI).
- Frameworks: the libraries for building RAG and agents (LangChain/LangGraph, LlamaIndex, CrewAI, AutoGen).
- Workflows: the visual platforms for orchestrating processing (Dify, n8n).
- Observability: collecting and analysing LLM traces (Langfuse, OpenLLMetry, Phoenix).
Two things tie these layers together: the OpenAI-compatible API, which most inference servers expose and most clients can consume, and OpenTelemetry, which lets you follow a request from one layer to the next.
The comparison table
| Building block | Layer | Strengths | Points to watch | Dedicated article |
|---|---|---|---|---|
| Ollama | Inference | One-command installation, model catalogue, simple API | Designed for a workstation or a small server; limited concurrency | Ollama |
| vLLM | Inference | High throughput on GPU, OpenAI-compatible API, Prometheus metrics | Requires GPUs and memory tuning | vLLM |
| llama.cpp | Inference | Lightweight, CPU or GPU, GGUF format, quantisation | Lower level; you have to industrialise it yourself | llama.cpp |
| Open WebUI | Interface | Complete chat interface, multi-user, built-in RAG | Project-specific licence, to be read; account management to be framed | Open WebUI |
| LangChain / LangGraph | Framework | Ecosystem of integrations, stateful agent graphs | Many abstractions, frequent changes | LangChain and LangGraph |
| LlamaIndex | Framework | RAG ingestion, indexing and evaluation | Less geared to orchestrating complex agents | LlamaIndex |
| CrewAI | Framework | Multi-agent by roles and tasks, very readable | Usage telemetry enabled by default | CrewAI and AutoGen |
| AutoGen | Framework | Conversational agents, native OpenTelemetry tracing | In maintenance mode, successor announced by Microsoft | CrewAI and AutoGen |
| Dify | Workflows | Visual LLM applications, RAG and agents, an API per application | Modified Apache 2.0 licence, with conditions | Dify and n8n |
| n8n | Workflows | Hundreds of connectors, webhooks, AI nodes | “Fair-code” licence, commercial restrictions | Dify and n8n |
| Langfuse | Observability | LLM traces, prompts, costs, evaluations; OTLP ingestion | Several components to operate when self-hosted | Langfuse and OpenLLMetry |
| OpenLLMetry | Observability | OpenTelemetry instrumentations for LLMs, vector databases and frameworks | gen_ai.* conventions still evolving |
Langfuse and OpenLLMetry |
| Phoenix | Observability | Traces and evaluations, based on OpenTelemetry and OpenInference | Elastic License 2.0 | AI observability |
This table summarises trends, not benchmarks: performance depends on the model, the hardware and the load. Measure in your own context.
Self-hosted AI infrastructure: how do you choose the inference layer?
This is the most far-reaching decision, because it determines the hardware:
- A workstation, a small team, experiments: Ollama, for its simplicity.
- A shared GPU service with many concurrent users: vLLM, designed for throughput and memory efficiency.
- Modest hardware, CPU, embedded or an isolated environment: llama.cpp and its built-in server, with quantised models.
Since all three expose an OpenAI-compatible API (Ollama also offers its own), switching servers later remains possible if clients rely on this common API. On Kubernetes, think about resource requests: our article on effective CPU requests for pods shows how wide the gap between declared and actual usage can be, and GPUs are even harder to share.
Frameworks or a workflow platform?
Frameworks (LangChain, LlamaIndex, CrewAI, AutoGen) are aimed at developers: everything is code, testable and versionable, but you have to write and maintain an application. Workflow platforms (Dify, n8n) are aimed at a wider audience: you assemble visually and publish an API, but you depend on the tool and its execution model.
A simple rule: if the core need is specific business processing with tests, use a framework; if it is integration between existing systems with an AI step, use a workflow platform. The two often coexist.
Why observability is not optional
A self-hosted AI stack combines the risks of a conventional service (outages, saturation, latency) with those specific to LLMs (wrong answers, agent loops, exploding token counts, data leaking through prompts). You therefore need two levels of observation:
- Infrastructure and service: GPU, memory, inference server queues, latency, errors, availability of each building block;
- LLM application: call traces, retrieved documents, tokens, quality scores.
OpenLLMetry or OpenInference produce the traces, Langfuse or Phoenix analyse them from an LLM angle, and a general-purpose OTLP-compatible backend ties everything to the rest of the system. Our pillar on AI observability details this approach, and the SRE requirements of AI workloads sets out its constraints.
In practice: three typical combinations
Step 1 — The starter kit (one machine)
Ollama + Open WebUI, optionally with Open WebUI’s built-in RAG. Goal: introduce the team to local AI and collect use cases. What to monitor: availability, memory, response time.
# indicative example: check that the Ollama server responds
curl http://localhost:11434/api/tags
Step 2 — The ops assistant (one GPU server)
vLLM for shared inference, Open WebUI as the interface, a LlamaIndex RAG on the runbooks exposed as an API, n8n for alert triage, and OpenLLMetry sending to Langfuse. Goal: a first internal service with an availability and latency SLO.
Step 3 — The internal AI platform (Kubernetes)
Several models served by vLLM behind a gateway, Dify for business teams, LangGraph or CrewAI for agents developed in-house, and a central OpenTelemetry Collector sending traces to Langfuse or Phoenix and to the company’s observability backend. Goal: a platform product, with a catalogue, quotas and cost tracking. See also our article on the challenges of platform engineering.
At each stage, add a building block only when a measured need justifies it.
Best practices
- Read the licences: several of the projects mentioned are not under a “classic” open source licence (Dify, n8n, Open WebUI, Phoenix).
- Standardise on the OpenAI-compatible API on the client side to keep the freedom to switch servers.
- Instrument with OpenTelemetry from the prototype stage, and centralise through a Collector.
- Pin image and model versions; test updates against a set of reference cases.
- Put every interface behind corporate authentication and a TLS reverse proxy.
- Expose tools to agents sparingly, possibly through MCP, and read-only by default.
- Track costs: GPUs, trace storage and the human time spent on operations.
Frequently asked questions
Is a self-hosted AI stack cheaper than a cloud API?
Not necessarily. It becomes worthwhile with sustained usage, confidentiality constraints or a need to work offline. You have to account for hardware, energy and above all operational time.
Which model should you start with?
A medium-sized general-purpose model, distributed under a licence compatible with your use, and available in your inference server’s format. Then compare a few candidates on your own questions rather than on public leaderboards.
Do you have to self-host everything, including observability?
It is not mandatory. Langfuse and Phoenix can be self-hosted, but traces can also go to a managed service if prompts are masked at the Collector. What matters is deciding consciously where the data goes.
Conclusion
A self-hosted open source AI stack is built in layers: inference, interface, frameworks, workflows and observability. There is no universal combination, but there is an approach: start small, standardise on open interfaces (OpenAI-compatible API, OpenTelemetry), measure, then add building blocks once the need is proven. The satellite articles in this series cover each of them in detail.
To go deeper into each project, refer to its official documentation, for example that of vLLM or Phoenix.
span.observer is an independent community blog, not affiliated with Dynatrace LLC or with the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.
4 comments