llama.cpp: local inference on CPU and GPU, GGUF models and the built-in HTTP server

Build llama.cpp, serve a GGUF model on CPU or GPU with the OpenAI-compatible llama-server, then monitor it: health checks, Prometheus metrics and logs.
llama.cpp : inférence locale sur CPU et GPU, modèles GGUF et serveur HTTP intégré

llama.cpp is the engine that brought local LLM inference to everyone: a C/C++ implementation with no heavy dependencies, able to run a language model on an ordinary CPU, a consumer GPU or a Mac. Started by Georgi Gerganov and now maintained under the ggml-org organisation, it is one of the most-followed open source projects on GitHub, and a building block for many tools, including Ollama.

Using llama.cpp directly rather than through a wrapper has several benefits: fine control over quantisation, context size and how layers are split between CPU and GPU, lightweight binaries that are easy to package, and a built-in HTTP server that speaks the OpenAI API dialect.

In this article we cover the key concepts (GGUF, quantisation, GPU offloading), build and start llama-server, then look at how to monitor it: health endpoint, Prometheus metrics, logs and OpenTelemetry traces on the application side, all the way to a backend such as Dynatrace.

llama.cpp in brief: GGUF, quantisation and backends

Three concepts are enough to get started.

  • GGUF is the model file format for llama.cpp: a single file holds the weights, the tokenizer and the metadata. Many open models are published in this format on Hugging Face.
  • Quantisation lowers the precision of the weights (8-bit, 5-bit, 4-bit…) to reduce the memory needed and speed up inference, at the cost of a slight loss of quality. Variants have names such as Q8_0 or Q4_K_M.
  • Backends decide which hardware is used: CPU by default, CUDA for NVIDIA, Metal on Apple Silicon, Vulkan, HIP for AMD, among others. You can offload all or some of the model’s layers to the GPU.

The project ships several tools: llama-cli to chat in the terminal, llama-server to serve a model over HTTP, llama-quantize to produce quantised variants, and llama-bench to measure performance on your own hardware.

In practice: local inference with llama.cpp

Step 1 — Install or build

The quickest route is a package (Homebrew, winget…) or the prebuilt binaries published with every release on GitHub. To enable a specific backend, build with CMake:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build                  # CPU only
# cmake -B build -DGGML_CUDA=ON # with CUDA
cmake --build build --config Release -j
ls build/bin/                   # llama-cli, llama-server, ...

Build options and the prerequisites for each backend are described in the build guide in the official llama.cpp repository.

Step 2 — Get a GGUF model

You can download a .gguf file from Hugging Face, or let llama.cpp do it with the -hf option, which takes a Hugging Face repository (and optionally a quantisation variant):

build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

Pick the quantisation according to the memory available: a 7 to 8 billion parameter model in 4-bit fits in a few gigabytes, whereas the 16-bit version needs several times more.

Step 3 — Start the HTTP server

build/bin/llama-server \
  -m models/my-model-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 8192 \
  -ngl 99 \
  --metrics

-c sets the context size, -ngl the number of layers offloaded to the GPU (a high value puts them all there if memory allows), and --metrics enables the Prometheus endpoint. An --api-key option protects access, and a parallelism option sets how many requests are served at once. Option names change over time: llama-server --help is the authority.

The server also offers a small web interface at its root, handy for a quick test. An official container image is published on the project’s GitHub registry.

Step 4 — Call the OpenAI-compatible API

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Give three causes of network latency."}]}'

Responses include token counts and, on the native endpoint, timing information for prompt processing and generation.

How do you monitor llama-server?

Health and availability

The /health endpoint tells you whether the server is ready; it returns an error while the model is still loading. It is the natural target for a Kubernetes readiness probe or a synthetic monitor.

Prometheus metrics

With --metrics, the /metrics path exposes metrics prefixed with llamacpp:: counters for prompt tokens and generated tokens, average throughput, requests being processed and requests deferred for lack of a free slot, plus the average number of busy slots per decode call and the highest context size observed. Query your instance to see the exact list:

curl -s http://127.0.0.1:8080/metrics | grep '^llamacpp'

Deferred requests are the saturation indicator to watch first: they mean every slot is busy and clients are waiting.

Logs

llama-server writes to standard output: hardware detection, model loading, how layers are split between CPU and GPU, then one timings line per request (prompt processing time, tokens per second during generation). These lines are invaluable for understanding a degradation; an OpenTelemetry Collector or a host agent can centralise them.

OpenTelemetry traces

To our knowledge, the server does not emit OpenTelemetry traces. Since it is OpenAI-compatible, the simplest approach is to instrument the calling application with an OpenAI instrumentation for OpenTelemetry (for example those from OpenLLMetry), with the client pointed at the llama-server URL. Each call becomes a span carrying the model, the tokens and the duration, attached to the trace of the user request.

Sending llama.cpp data to Dynatrace

The pipeline is the same as for vLLM: an OpenTelemetry Collector scrapes /metrics with the prometheus receiver, converts the counters to delta, receives OTLP traces from the application and exports to the Dynatrace OTLP endpoint with an API token. We detail this configuration in vLLM in production; you only need to change the scrape target. On a host running OneAgent, process and GPU metrics complete the picture.

Useful indicators: generated tokens per second, deferred requests, client-side latency per request, GPU utilisation and memory, and number of restarts. To correlate this data with the rest of the chain, apply the principles from our article end-to-end OpenTelemetry.

Best practices

  • Benchmark with llama-bench on your hardware before choosing a quantisation.
  • Always enable --metrics and a probe on /health as soon as a server is shared.
  • Keep --host 127.0.0.1 by default and add an API key or a reverse proxy before exposing anything.
  • Pin the version: the project releases very often and options change.
  • Only download GGUF models from trusted sources.

Frequently asked questions

What is the difference between llama.cpp and Ollama?

Ollama is built on llama.cpp and adds simplified model management. llama.cpp gives you more control and more options, with a little more configuration. See our guide Ollama locally.

Does llama.cpp work without a GPU?

Yes, that is in fact where it started. Performance then depends heavily on the CPU, memory bandwidth and the quantisation chosen.

How do you enable llama-server’s Prometheus metrics?

Start the server with the --metrics option; the metrics are then exposed on /metrics, on the same port as the API.

Conclusion

llama.cpp remains the reference for lightweight local inference, from a laptop to a small GPU server. Its built-in server, OpenAI-compatible and equipped with Prometheus metrics, makes it easy to integrate and to observe. To see how it compares with Ollama, vLLM and the other building blocks, read our comparison of self-hosted open source AI stacks.

span.observer is an independent community blog, not affiliated with Dynatrace LLC or with the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.

Total
0
Shares
2 comments
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
vLLM production : serveur compatible OpenAI, métriques Prometheus et supervision des GPU

vLLM production metrics: running an OpenAI-compatible server with Prometheus and GPU monitoring

Next Post
Open WebUI auto-hébergé : un ChatGPT interne devant Ollama ou une API compatible OpenAI

Self-hosted Open WebUI: an internal ChatGPT in front of Ollama or an OpenAI-compatible API

Related Posts