Self-hosted open source AI stack: comparing the building blocks of a self-hosted AI infrastructure

Compare the building blocks of a self-hosted open source AI stack: Ollama, vLLM, Open WebUI, LlamaIndex, CrewAI, Dify, n8n, Langfuse and typical setups.
Stack IA open source auto-hébergée : le comparatif des briques d'une infrastructure IA auto-hébergée

Building a self-hosted open source AI stack is no longer reserved for research teams. In 2026, an ops team can run an LLM on its own servers, offer colleagues a chat interface, plug a RAG into its documentation, automate alert triage and trace every call, without any data leaving its infrastructure. This guide compares the building blocks of that self-hosted AI infrastructure, layer by layer.

The difficulty is no longer finding tools but choosing between them. Ollama, vLLM, llama.cpp, Open WebUI, LangChain, LlamaIndex, CrewAI, Dify, n8n, Langfuse, Phoenix: every project has its place, but not necessarily in your context. And every building block you add is one more service to deploy, update, secure and monitor.

This pillar guide organises these projects into five layers, compares their strengths and limits in a table, then suggests typical combinations. Each building block has a dedicated article, linked below, with a step-by-step guide and operational advice.

The five layers of a self-hosted open source AI stack

  1. Inference: the server that loads the model and answers requests (Ollama, vLLM, llama.cpp).
  2. Interface: the chat application for users (Open WebUI).
  3. Frameworks: the libraries for building RAG and agents (LangChain/LangGraph, LlamaIndex, CrewAI, AutoGen).
  4. Workflows: the visual platforms for orchestrating processing (Dify, n8n).
  5. Observability: collecting and analysing LLM traces (Langfuse, OpenLLMetry, Phoenix).

Two things tie these layers together: the OpenAI-compatible API, which most inference servers expose and most clients can consume, and OpenTelemetry, which lets you follow a request from one layer to the next.

The comparison table

Building block Layer Strengths Points to watch Dedicated article
Ollama Inference One-command installation, model catalogue, simple API Designed for a workstation or a small server; limited concurrency Ollama
vLLM Inference High throughput on GPU, OpenAI-compatible API, Prometheus metrics Requires GPUs and memory tuning vLLM
llama.cpp Inference Lightweight, CPU or GPU, GGUF format, quantisation Lower level; you have to industrialise it yourself llama.cpp
Open WebUI Interface Complete chat interface, multi-user, built-in RAG Project-specific licence, to be read; account management to be framed Open WebUI
LangChain / LangGraph Framework Ecosystem of integrations, stateful agent graphs Many abstractions, frequent changes LangChain and LangGraph
LlamaIndex Framework RAG ingestion, indexing and evaluation Less geared to orchestrating complex agents LlamaIndex
CrewAI Framework Multi-agent by roles and tasks, very readable Usage telemetry enabled by default CrewAI and AutoGen
AutoGen Framework Conversational agents, native OpenTelemetry tracing In maintenance mode, successor announced by Microsoft CrewAI and AutoGen
Dify Workflows Visual LLM applications, RAG and agents, an API per application Modified Apache 2.0 licence, with conditions Dify and n8n
n8n Workflows Hundreds of connectors, webhooks, AI nodes “Fair-code” licence, commercial restrictions Dify and n8n
Langfuse Observability LLM traces, prompts, costs, evaluations; OTLP ingestion Several components to operate when self-hosted Langfuse and OpenLLMetry
OpenLLMetry Observability OpenTelemetry instrumentations for LLMs, vector databases and frameworks gen_ai.* conventions still evolving Langfuse and OpenLLMetry
Phoenix Observability Traces and evaluations, based on OpenTelemetry and OpenInference Elastic License 2.0 AI observability

This table summarises trends, not benchmarks: performance depends on the model, the hardware and the load. Measure in your own context.

Self-hosted AI infrastructure: how do you choose the inference layer?

This is the most far-reaching decision, because it determines the hardware:

  • A workstation, a small team, experiments: Ollama, for its simplicity.
  • A shared GPU service with many concurrent users: vLLM, designed for throughput and memory efficiency.
  • Modest hardware, CPU, embedded or an isolated environment: llama.cpp and its built-in server, with quantised models.

Since all three expose an OpenAI-compatible API (Ollama also offers its own), switching servers later remains possible if clients rely on this common API. On Kubernetes, think about resource requests: our article on effective CPU requests for pods shows how wide the gap between declared and actual usage can be, and GPUs are even harder to share.

Frameworks or a workflow platform?

Frameworks (LangChain, LlamaIndex, CrewAI, AutoGen) are aimed at developers: everything is code, testable and versionable, but you have to write and maintain an application. Workflow platforms (Dify, n8n) are aimed at a wider audience: you assemble visually and publish an API, but you depend on the tool and its execution model.

A simple rule: if the core need is specific business processing with tests, use a framework; if it is integration between existing systems with an AI step, use a workflow platform. The two often coexist.

Why observability is not optional

A self-hosted AI stack combines the risks of a conventional service (outages, saturation, latency) with those specific to LLMs (wrong answers, agent loops, exploding token counts, data leaking through prompts). You therefore need two levels of observation:

  • Infrastructure and service: GPU, memory, inference server queues, latency, errors, availability of each building block;
  • LLM application: call traces, retrieved documents, tokens, quality scores.

OpenLLMetry or OpenInference produce the traces, Langfuse or Phoenix analyse them from an LLM angle, and a general-purpose OTLP-compatible backend ties everything to the rest of the system. Our pillar on AI observability details this approach, and the SRE requirements of AI workloads sets out its constraints.

In practice: three typical combinations

Step 1 — The starter kit (one machine)

Ollama + Open WebUI, optionally with Open WebUI’s built-in RAG. Goal: introduce the team to local AI and collect use cases. What to monitor: availability, memory, response time.

# indicative example: check that the Ollama server responds
curl http://localhost:11434/api/tags

Step 2 — The ops assistant (one GPU server)

vLLM for shared inference, Open WebUI as the interface, a LlamaIndex RAG on the runbooks exposed as an API, n8n for alert triage, and OpenLLMetry sending to Langfuse. Goal: a first internal service with an availability and latency SLO.

Step 3 — The internal AI platform (Kubernetes)

Several models served by vLLM behind a gateway, Dify for business teams, LangGraph or CrewAI for agents developed in-house, and a central OpenTelemetry Collector sending traces to Langfuse or Phoenix and to the company’s observability backend. Goal: a platform product, with a catalogue, quotas and cost tracking. See also our article on the challenges of platform engineering.

At each stage, add a building block only when a measured need justifies it.

Best practices

  • Read the licences: several of the projects mentioned are not under a “classic” open source licence (Dify, n8n, Open WebUI, Phoenix).
  • Standardise on the OpenAI-compatible API on the client side to keep the freedom to switch servers.
  • Instrument with OpenTelemetry from the prototype stage, and centralise through a Collector.
  • Pin image and model versions; test updates against a set of reference cases.
  • Put every interface behind corporate authentication and a TLS reverse proxy.
  • Expose tools to agents sparingly, possibly through MCP, and read-only by default.
  • Track costs: GPUs, trace storage and the human time spent on operations.

Frequently asked questions

Is a self-hosted AI stack cheaper than a cloud API?

Not necessarily. It becomes worthwhile with sustained usage, confidentiality constraints or a need to work offline. You have to account for hardware, energy and above all operational time.

Which model should you start with?

A medium-sized general-purpose model, distributed under a licence compatible with your use, and available in your inference server’s format. Then compare a few candidates on your own questions rather than on public leaderboards.

Do you have to self-host everything, including observability?

It is not mandatory. Langfuse and Phoenix can be self-hosted, but traces can also go to a managed service if prompts are masked at the Collector. What matters is deciding consciously where the data goes.

Conclusion

A self-hosted open source AI stack is built in layers: inference, interface, frameworks, workflows and observability. There is no universal combination, but there is an approach: start small, standardise on open interfaces (OpenAI-compatible API, OpenTelemetry), measure, then add building blocks once the need is proven. The satellite articles in this series cover each of them in detail.

To go deeper into each project, refer to its official documentation, for example that of vLLM or Phoenix.

span.observer is an independent community blog, not affiliated with Dynatrace LLC or with the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.

Total
0
Shares
4 comments
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
CrewAI/AutoGen : construire des systèmes multi-agents, et l'observabilité CrewAI avec OpenTelemetry

CrewAI + AutoGen: building multi-agent systems, and CrewAI observability with OpenTelemetry

Next Post
Formation Dynatrace Managed : le parcours complet en 8 modules

Dynatrace Managed training: a free Dynatrace course in 8 modules

Related Posts