LLM-as-judge in production means having a second language model score the real answers of a generative AI application. The idea is attractive: an automatic score replaces human review. But each score is itself a model call, so it adds cost and latency. Without limits, the judge can cost as much as the application it watches.
This article offers a method to keep that cost under control: score a sample, target the right steps, calibrate the judge on human examples, then monitor the judge like any other service. It complements the article on evaluating a RAG with LlamaIndex, which mostly covers offline evaluation. Here, the angle is continuous operation from an SRE point of view.
In brief
- An LLM judge receives a rubric, the input, the output and, if needed, a reference; it returns a score and a justification.
- In production, the budget is set by sampling, by targeting observations and by the choice of judge model.
- A judge has documented biases: calibrate it on labelled examples before relying on it.
- The judge itself must be observed: cost, duration, errors, queue.
LLM-as-judge in production: what a judge costs
According to the Langfuse documentation, a judge receives an evaluation rubric, the input, the output to evaluate and an optional reference. It returns a score with its reasoning. The same page states that cost depends on the judge model and the size of the inputs. Two levers follow: fewer scores, and less volume sent to the judge.
The difference from offline evaluation is simple. Offline, a fixed test set is replayed before a release. In production, traffic is open-ended, large and always changing. Scoring every request is neither necessary nor desirable. So you must decide what to score, how often and with which model.
Choosing what to score
The Langfuse documentation describes a live evaluation rule as a set of filters, a sampling rate and one or more evaluator assignments. This breakdown is useful even outside that tool, because it asks the right questions.

Target the step that matters
Langfuse recommends observation-level evaluators for production data and says trace-level ones are deprecated. In practice, score the model’s final answer, not every intermediate call of an agent. The page adds that an observation-level evaluator does not load sibling or child observations, so all useful information must be in the evaluated input and output.
Filter before sampling
Available filters cover the observation’s type, name and metadata, plus the trace’s user, session, tags and version. A version filter is valuable: after a prompt or model change, score the new version more heavily, then ease off once confidence returns. For how traces and identifiers work, see the article on Langfuse and OpenLLMetry.
Sample in a reproducible way
Pure random draws make comparisons hard. Deterministic sampling on the trace ID guarantees that a given trace is always scored, or never, whichever service decides. The example below is a teaching sketch from span.observer, not a library function.
import hashlib
def should_score(trace_id: str, rate: float) -> bool:
# True for about `rate` of traces, in a stable way.
h = int(hashlib.sha256(trace_id.encode()).hexdigest(), 16)
return (h % 10_000) < int(rate * 10_000)
# 5% of normal traffic, 100% of flagged traces (error, negative feedback)
rate = 1.0 if is_flagged else 0.05
if should_score(trace_id, rate):
send_to_judge(trace_id)
Two strata are often enough: a small percentage of ordinary traffic, and all already suspicious cases (negative feedback, errors, recent version). The reasoning is the same as for trace sampling: keep what is rare and useful.
Calibrate the judge before trusting it
A judge is not a neutral instrument. The research paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” names four weaknesses: sensitivity to answer order, a preference for longer answers, favouritism towards its own outputs and limited reasoning. Its authors also report that a strong judge such as GPT-4 reaches over 80% agreement with humans, the same level as agreement between humans. That figure comes from their study and datasets. It does not carry over to your rubric or your answers.

The Langfuse page gives the same practical advice: check the judge prompt against labelled examples before using it. Here is a short method.
- Build a golden set: a few dozen real answers, labelled by someone who knows the domain.
- Write a narrow rubric: one criterion per judge (faithfulness to context, for example), with a boolean or categorical score rather than a mark out of ten.
- Compare the judge’s scores with the human labels, and read the disagreements one by one.
- Fix the rubric, then repeat. Redo this check whenever the judge model changes.
- Review a sample every week: traffic drift eventually misaligns the judge.
Categorical and boolean scores, offered by Langfuse alongside numeric ones, are easier to calibrate: a disagreement is clear-cut. According to Langfuse’s data model, a score attaches to a trace, an observation, a session or a dataset run.
Cutting the price of each score
- A cheaper judge when the rubric is simple. The Langfuse page says an evaluator uses the project’s default model unless a dedicated one is chosen. Test a smaller model on the golden set and keep it if it agrees closely enough with humans.
- Short inputs: send only the question, the answer and the context passages actually used.
- Deterministic checks first: valid JSON, length, presence of a citation. Whatever code can test needs no judge.
- A trigger threshold: score only the answers that passed the simple checks.
To tie these choices to tokens consumed, the article on AI agent cost shows how to track tokens with OpenTelemetry.
Observing the judge itself
A failing judge produces an empty dashboard, which is wrongly read as good news. Langfuse runs evaluators in its worker, not in the application. Each execution creates a trace: the documentation says to filter the environment langfuse-llm-as-a-judge in the tracing table. The statuses are Completed, Error, Delayed (rate-limited, retrying) and Pending. That page also notes that execution traces let you monitor token usage.
Four signals to follow, whatever the tool:
- the number of scores produced per hour, compared with the expected sampling rate;
- the share of executions in error or pending;
- tokens consumed by the judge, separate from the application’s;
- the score distribution: a sudden shift often points to a prompt or model change.
If the judge is called through an OpenTelemetry-instrumented SDK, the call duration can feed the gen_ai.client.operation.duration histogram (unit s). According to the OpenTelemetry GenAI specification, this metric still has Development status: its names may change. This post describes no documented Dynatrace integration for judge scores. A team using Dynatrace can publish its scores as attributes or metrics through OpenTelemetry, provided it defines the schema itself. AI agent guardrails complement this monitoring.
Best practices
- One judge per criterion, with a short rubric and categorical outputs.
- A human golden set, reviewed at every model or prompt change.
- An explicit budget: scores per day and maximum tokens, with an alert on overrun.
- A judge from a different family than the model being evaluated, to limit favouritism towards its own outputs.
- Scores help spot trends and direct human review; they should not decide a block on their own.
Frequently asked questions
Should 100% of answers be scored in production?
Rarely. A stable sample of ordinary traffic, plus all suspicious cases, gives a usable signal at a bounded cost. Langfuse offers a sampling rate in each evaluation rule.
Can an LLM judge replace human evaluation?
No. It extends it. The MT-Bench authors report strong agreement with humans for a powerful judge, but also describe biases. Regular human review remains necessary to keep the judge aligned.
How can I tell whether the judge costs too much?
Measure the judge’s tokens and calls separately from the application, then relate them to trace volume. If the judge approaches the application’s cost, lower the sampling rate, shorten the inputs or test a smaller model.
Conclusion
An LLM-as-judge in production remains a service in its own right, with a budget, errors and possible drift. Sampling, targeting the right step and calibrating on a human set give a useful signal at a controlled cost.
Sources
- LLM-as-a-Judge, Langfuse documentation
- Evaluation data model, Langfuse documentation
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv
- OpenTelemetry GenAI semantic conventions
span.observer is an independent blog, not affiliated with Dynatrace LLC or Langfuse. Product names belong to their respective owners; the code is a teaching example.