Browsing Tag
llama.cpp
3 posts
Ollama vLLM llama.cpp: which local engine should you choose for LLMs?
Ollama vLLM llama.cpp: compare these three local inference engines by use case, hardware and monitoring (metrics, health checks, traces).
Jan open source ChatGPT alternative: offline local AI
Jan open source ChatGPT alternative: offline local AI, an OpenAI-compatible server on localhost:1337, installation, limits and monitoring.
llama.cpp: local inference on CPU and GPU, GGUF models and the built-in HTTP server
Build llama.cpp, serve a GGUF model on CPU or GPU with the OpenAI-compatible llama-server, then monitor it: health checks, Prometheus metrics and logs.