Artificial intelligence agents are no longer science experiments confined to research labs. They are booking flights, writing code, answering customer questions, and making decisions that affect real people and real money. But as these agents move into production, a critical gap has emerged: we can monitor them, but we cannot easily test them where it matters most, in the messy, unpredictable real world. Datadog’s move to bring production experimentation to AI agent observability is a direct response to that gap. It signals a major shift in how businesses will build, deploy, and trust AI agents going forward.
AI agents are software systems that can reason, plan, and take actions to complete tasks. Unlike traditional chatbots that respond with text, agents can call tools, access databases, send emails, and chain together multiple steps to achieve a goal. This makes them incredibly powerful, and incredibly hard to manage.
Traditional software is deterministic. Given the same input, it produces the same output. AI agents are not. Their behavior depends on the model, the prompt, the context, the tools available, and even the time of day. Two identical requests can produce different results. This non-determinism is what makes agents useful for complex tasks, but it is also what makes them risky to deploy at scale.
Most companies today test their AI agents in staging environments with curated datasets. That works for catching obvious bugs, but it fails to capture the full range of real-world inputs. Production traffic contains edge cases, adversarial inputs, and shifting user behavior that no test set can fully anticipate. The result is a uncomfortable truth: many AI agents are not truly tested until they are live, facing real users and real consequences.
Production experimentation is the practice of testing changes with a controlled subset of live traffic before rolling them out broadly. It includes techniques like A/B testing, canary releases, and shadow mode, where a new version runs alongside the old one without affecting users. For traditional software, these techniques are well understood. For AI agents, they are essential, and largely missing.
Consider a customer support agent that handles refund requests. A new version of the agent might use a different language model or a new prompt. In staging, it performs well on a test set of 500 examples. But in production, it encounters angry customers, ambiguous requests, and attempts to game the system. Without production experimentation, the company either ships the change blindly or delays it indefinitely. Neither is acceptable.
Production experimentation lets teams answer critical questions safely: Does the new agent version improve task success rates? Does it reduce cost per interaction? Does it introduce new safety risks? Does it perform well across different customer segments? These questions cannot be answered with static tests alone. They require live traffic, controlled rollout, and rigorous measurement.
Datadog is a major player in observability, the practice of monitoring software systems in production. Its platform already helps teams track performance, errors, and logs across their infrastructure. By bringing production experimentation to AI agent observability, Datadog is closing the loop between seeing what agents do and improving what they do.
This is a significant strategic move. It acknowledges that observability alone is not enough for AI agents. You can monitor an agent’s latency and error rates, but that tells you nothing about whether the agent is making good decisions. You need to connect observability data to experimentation frameworks so you can test changes, measure outcomes, and iterate quickly.
The move also signals that the market for AI agent observability is heating up. As more companies deploy agents, the demand for tools that provide visibility, control, and safety will grow. Datadog’s entry into production experimentation suggests that the next battleground is not just monitoring, it is continuous improvement through controlled testing.
While the specific implementation details are still emerging, the general approach to production experimentation for AI agents follows a recognizable pattern. Teams instrument their agents to emit detailed traces, records of every step, tool call, and decision. They define key metrics such as task success rate, cost per task, latency, and safety violations. Then they use traffic segmentation to route a small percentage of users to a new agent version while the majority continue using the current version.
During the experiment, the system collects data on both versions. Teams compare the metrics to see if the new version is better, worse, or equivalent. If it is better, they gradually increase traffic to the new version. If it is worse or introduces risks, they roll back. This cycle can repeat continuously, allowing teams to improve agents week by week rather than in big, risky releases.
Shadow mode is particularly valuable for high-stakes agents. In shadow mode, the new agent version processes live traffic but its actions are not shown to users. Instead, its outputs are logged and compared to the current version. This allows teams to catch failures before they affect anyone. For example, a financial agent might be tested in shadow mode to see if it makes risky investment recommendations. Only after passing shadow tests would it be gradually exposed to real users.
The arrival of production experimentation for AI agents has several practical implications. First, businesses need to invest in instrumentation. You cannot experiment on what you cannot measure. Agents must be built with observability in mind, emitting structured traces and metrics from day one.
Second, teams need to define clear evaluation criteria. What does success look like for your agent? Is it task completion? Customer satisfaction? Cost efficiency? Safety? These metrics should be agreed upon before experimentation begins, not after.
Third, organizations should adopt a culture of continuous experimentation. AI agents are not “set and forget” systems. They require ongoing tuning as user behavior changes, models improve, and new risks emerge. Companies that treat agent deployment as a one-time event will fall behind those that treat it as an ongoing process.
Fourth, cross-functional collaboration becomes essential. Experimentation involves data scientists, engineers, product managers, and compliance teams. Each group brings a different perspective on what to test and how to interpret results. Siloed teams will struggle to run effective experiments.
Finally, cost management is critical. AI agents can be expensive to run, especially when they use large language models. Production experimentation helps teams identify cost-saving opportunities, for example, by testing a cheaper model on a subset of traffic to see if it maintains quality. Without experimentation, companies risk overspending on inefficient agent designs.
Looking ahead, production experimentation for AI agents is likely to become more automated. Imagine an agent that automatically runs experiments on itself, testing new prompts or tool configurations against live traffic and rolling out improvements without human intervention. This is the frontier of self-improving agents, systems that learn from production data and evolve over time.
Such systems raise important governance questions. Who is responsible when an agent makes a bad decision? How do you audit an agent that changes its own behavior? Production experimentation provides a foundation for answering these questions by creating a record of what was tested, what was rolled out, and what the outcomes were. This audit trail will be essential for regulatory compliance and for building trust with users.
We are also likely to see experimentation extend beyond individual agents to multi-agent systems. As agents begin collaborating with each other, testing becomes more complex. A change to one agent can affect the behavior of others. Production experimentation will need to account for these interactions, testing not just individual agents but entire agent ecosystems.
Datadog’s move to bring production experimentation to AI agent observability is more than a product update. It is a recognition that AI agents are becoming first-class citizens in the software stack, deserving the same rigor and discipline we apply to any critical system. The companies that embrace this discipline will be the ones that deploy agents safely, efficiently, and at scale. Those that don’t will learn the hard way that trust, once lost, is hard to rebuild.