CrewAI and AutoGen are two open source frameworks for building multi-agent systems: several specialised agents, each with its own role, tools and sometimes its own model, working together to complete a task. For an ops team the idea is appealing: one agent reads the alerts, another queries the logs, a third writes the summary. Which tool to pick is the purpose of this CrewAI/AutoGen comparison, which ends with CrewAI observability: how to trace what each agent does.
But a multi-agent system also multiplies the sources of uncertainty. Agents exchange messages, call tools and call the model again, and a single task can trigger dozens of calls. Without fine-grained observation, there is no way to know why a result is wrong, why the task took two minutes or why the token bill tripled.
This article introduces both frameworks, their mental model and a concrete example for each, then shows how to observe them with OpenTelemetry. It is part of our comparison of the self-hosted open source AI stack.
CrewAI + AutoGen: two mental models
CrewAI: a team with roles and tasks
CrewAI is a Python framework that organises work like a team. You define agents (a role, a goal, a context called the backstory, tools), tasks (a description and an expected output, assigned to an agent) and a crew that runs them according to a process: sequential, or hierarchical with a manager agent. CrewAI also offers Flows, for chaining steps in a more deterministic, event-driven way.
This model is very readable: you describe an organisation, not a graph. It suits processing chains whose steps are known in advance.
AutoGen: agents that converse
AutoGen, from Microsoft Research, models collaboration as a conversation. Since its rewrite (v0.4), it is organised in layers: autogen-core (a message-driven agent runtime), autogen-agentchat (the high-level API: AssistantAgent, teams such as RoundRobinGroupChat or SelectorGroupChat, termination conditions) and autogen-ext (model clients, tools, MCP).
An important point before starting a project: the AutoGen repository now states that the project is in maintenance mode and directs new users to Microsoft Agent Framework, presented as its successor, with a migration guide. AutoGen remains usable and instructive, but for new development, evaluate that successor. There is also AG2, a community project derived from AutoGen’s original branch.
When is a multi-agent system justified?
Often a single agent with good tools is enough, or even a deterministic workflow. Multiple agents are justified when:
- the sub-tasks need very different contexts or prompts;
- you want cross-review (one agent writes, another critiques);
- you want to restrict the tools available to each agent, for security reasons.
Every extra agent adds latency, cost and points of failure. Start simple, then measure.
In practice: two incident analysis agents and CrewAI observability
Step 1 — With CrewAI
Install with pip install crewai. For a local model via Ollama, CrewAI relies on LiteLLM, which may need an extra dependency depending on the version (see the CrewAI documentation).
from crewai import Agent, Task, Crew, Process, LLM
llm = LLM(model="ollama/llama3.1", base_url="http://localhost:11434")
analyste = Agent(
role="Incident analyst",
goal="Identify the probable cause from the logs provided",
backstory="Experienced SRE, factual, quotes the log lines.",
llm=llm)
redacteur = Agent(
role="Writer",
goal="Produce a clear summary for the on-call team",
backstory="Writes short, actionable reports.",
llm=llm)
analyse = Task(description="Analyse these logs: {logs}",
expected_output="Probable cause and evidence", agent=analyste)
synthese = Task(description="Write the incident summary",
expected_output="5 lines maximum", agent=redacteur)
crew = Crew(agents=[analyste, redacteur], tasks=[analyse, synthese],
process=Process.sequential)
print(crew.kickoff(inputs={"logs": extrait_logs}))
Step 2 — With AutoGen AgentChat
Install with pip install -U "autogen-agentchat" "autogen-ext[ollama]".
import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import MaxMessageTermination
from autogen_ext.models.ollama import OllamaChatCompletionClient
async def main():
client = OllamaChatCompletionClient(model="llama3.1")
analyste = AssistantAgent("analyste", model_client=client,
system_message="Identify the probable cause in the logs.")
relecteur = AssistantAgent("relecteur", model_client=client,
system_message="Critique the analysis and flag unproven hypotheses.")
equipe = RoundRobinGroupChat([analyste, relecteur],
termination_condition=MaxMessageTermination(6))
print(await equipe.run(task="Logs: ..."))
await client.close()
asyncio.run(main())
Note the termination condition: without it, a conversation between agents can run far longer than expected.
Step 3 — Enable OpenTelemetry traces
- AutoGen: the documentation describes native tracing. You just configure a global OpenTelemetry
TracerProvider(with an OTLP exporter) before creating the agents; you can add the model client instrumentations to it. - CrewAI: OpenLLMetry provides a CrewAI instrumentation, and the CrewAI documentation lists several observability integrations (Langfuse, Arize Phoenix, OpenLIT, etc.).
The OpenLLMetry, Collector and backend chain is explained in detail in Langfuse and OpenLLMetry. The principles match those of monitoring MCP agents, and the agents’ spans can attach to the calling service’s trace as described in end-to-end OpenTelemetry.
Note: by default CrewAI collects anonymous usage telemetry and sends it to its maintainers; its README says it can be disabled with OTEL_SDK_DISABLED=true. Be careful: this variable disables the OpenTelemetry SDK for the whole process, and therefore your own traces too. Check the interaction in the documentation before using it.
Step 4 — Choose the indicators
- number of model calls and conversation turns per task;
- tokens consumed per agent and per task;
- latency per agent and per tool call;
- share of tasks stopped by the guard condition rather than completing normally;
- tool errors and retries.
These measurements then feed SLOs as for any other service; see AI and SRE reliability.
Best practices
- Always set a limit on turns, duration or tokens per task.
- Give each agent only the tools it needs, read-only by default.
- Require structured outputs between agents to limit context drift.
- Keep a human in the loop for any action on production.
- Replay a set of reference cases after every prompt or model change.
- Name agents and tasks clearly: these names are what will appear in the traces.
Frequently asked questions
CrewAI vs AutoGen: which should you choose?
CrewAI for processes with known steps, described as a team with roles. AutoGen for open-ended conversations between agents, bearing in mind its move to maintenance mode and its announced successor.
Do these frameworks work with local models?
Yes, both can use Ollama or an OpenAI-compatible server. Small models, however, follow role and tool-calling instructions less well: test before rolling out widely.
How do you spot an agent stuck in a loop?
In the traces: an abnormal number of turns, repeated tool calls with the same arguments, a sharp rise in tokens. An alert on these indicators catches the problem before the bill does.
Conclusion
CrewAI and AutoGen make multi-agent systems accessible, with two different philosophies: the organised team and the conversation. In both cases, their value in production depends on your ability to observe what the agents do, to bound their behaviour and to measure their cost. Instrument from the very first prototype.
span.observer is an independent community blog, not affiliated with Dynatrace LLC or with the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.
1 comment