THE LINUX FOUNDATION PROJECTS
Community Blog

Nondeterminism in Agentic AI applications

By November 5, 2026No Comments

This blog post was developed by the Exploration Workstream of the Generative AI Commons which is a Working Group in LF AI & Data. The Exploration Workstream has a broad charter allowing it to explore new areas related to AI, research them, and publish its findings. The goal is to provide the community with an opportunity to investigate developments beyond what is currently mainstream and inform the LF AI & Data community on topics it may want to consider for future work. Non determinism in agentic AI applications has been a key topic of exploration.

Autonomous systems working on the principles of agentic AI today boast of achieving complex multi-step problem-solving, real time tool access, and memory based context-awareness beyond passive chat capabilities. In this new world of agentic systems, autonomous behavior falls along a spectrum from completely deterministic to highly non-deterministic. Many applications prefer deterministic behavior in AI systems where the autonomous process is controlled and reproducible while non-deterministic behavior today trades this predictability for enhanced flexibility or speed of execution in cases when the problem space is large and a deterministic solution is computationally intractable. To understand the concept, let’s take an example, it can be observed that the exact same question to a real time AI chatbot can provide slightly different answers each time making the output nondeterministic. Such adaptive behavior may be encouraged in many applications like product recommendations, user Q&A and content summarization. But other highly regulated domain use cases such as healthcare scheduling, legal and financial disclosures etc need highly deterministic and reproducible workflows. This is a very different challenge from the non agentic software applications and there are many reasons for such nondeterministic behavior.

Model Inference Process 

The output from an agentic application could largely depend on the underlying language model and inference process. A result from a language model involves “sampling”, a process that converts the language model’s prediction into a probability distribution and then selects the next token. Various sampling techniques are employed in agentic applications to break repetitiveness in terms of the token generation process and thus benefit from randomness during inference.

At a fundamental level, we can observe non-determinism in digital computing as a result of floating point non-associativity. While every forward pass action in most language models is deterministic i.e are not affected by the floating point non-associativity, the output token generation process as implemented by most open source inference engines depends on the inference batch sizes during forward passes. Thus in practice, even when making the sampling theoretically deterministic from an end user point of view, by using commonly used sampling parameters like temperature, the LLM APIs are still not deterministic in practice due to the underlying kernels not being batch invariant. There are implementations of batch invariant kernels available to force deterministic outputs on various open source inference engines but there are tradeoffs in inference latency and hence that may not be the default behavior.

Evolution of agentic components

Another reason for non-determinism stems from the underlying agentic components that may change or evolve over time. For example, software coding assistants automatically roll out access to new versions of underlying models with superior capabilities almost every month. Similarly, these agents may get access to different versions of newer tools. Memory capability or guardrails. Even in an enterprise setting, model context protocol (MCP) servers for various tools may have a versioned rollout that will modify the context and responses utilized by the many enterprise agents that depend on these tools. Such evolution of agentic components may lead to change in context structure and styles that may be invisible to the end user of these modern AI applications. Such changes may result in a difference in application behaviour and the degree of non determinism depends on the ability of architecture to withstand changes in its agentic execution environment.

Context in multi-agent systems

Most modern AI apps are multi-agent systems that coordinate specialized components to tackle complex workflows. Multi-agent patterns are becoming increasingly popular due to its value when a single agent has too many components and the underlying model capability is deemed insufficient to make accurate decisions about tool choice or task execution. At the center of multi-agent design is context engineering i.e deciding what context is passed to each subagent and how the response from each agent is finally synthesized. This nuance of context engineering establishes the degree of determinism in modern language model based agentic applications. The manner in which the context is presented, ordered for various sub agents and summarized can make all the difference in the application outputs from an end user perspective. This context could involve responses from various sub-agents, tool calls and other relevant information from past memory. Naturally, as the responses from various realtime tools change, the model output is expected to change, just like from a search engine today. But even if the tool responses remain exactly the same, the structure of context can change based on the order of subagent execution, sub-agent parallelization strategies employed and context synthesis processes to create a final output. For example, during an agentic workflow execution, if any one subagent responds to a query slightly slower than usual due to an intermittent network issue, the context generated for the next inference step in the agentic workflow could be very different than usual depending on the execution and context engineering pattern in play. This can manifest as nondeterministic behavior in AI applications.

To sum up, there are many reasons for non deterministic behaviour in various layers of modern agentic applications and a deeper evaluation of each layer and adaptable configurations of these systems is required  if reproducibility needs to be ensured for any real world application. 

About the Authors

Sachin Mathew Varghese is an AI Engineering Leader and open source advocate with a vast experience building software products in the domain of machine learning and AI operations mainly focused on model serving at scale, inference optimizations, deployment observability and enterprise management operations. Over the last decade, he has built AI solutions for technology organizations across the United Kingdom, India and the United States. Sachin is a maintainer on Kubernetes SIG project for GenAI inference performance benchmarking tool. He also leads the Exploration workstream at the Linux Foundation AI & Data Generative AI Commons promoting the adoption and advancement of secure, open and efficient AI models and applications architectures.

Arnaud Le Hors is Senior Technical Staff Member of Open Technologies at IBM, working on a range of technologies with a primary focus on Open source AI and security. He has been working on standards and open source development for over 30 years. He currently is the main representative for IBM at W3C and INCITS, a member of the OpenSSF Technical Advisory Committee, and co-chair of the LF AI & Data Generative Commons.

Dr. Deborah Dahl has over 30 years of experience in natural language and speech processing technology. She has developed many natural language systems for customers including large corporations, startups, and government agencies. She is also an expert on speech and language standards, with over 20 years of experience in standards at the World Wide Web Consortium, where she is a W3C Invited Expert, and the Linux Foundation AI and Data Foundation. At the Linux Foundation, Dr. Dahl is the project lead for the Open Voice Interoperability initiative which is developing standards for interoperable conversational assistants. Dr. Dahl’s most recent book is Natural Language Understanding with Python, published in 2023.

Cheranellore (Vasu) Vasudevan is a technology advisor and inventor in Artificial Intelligence (AI) and data science disciplines. He retired from IBM after a long professional tenure in research, product development, and technology strategy. He has a Ph. D. in Artificial Intelligence (AI) and several journal and conference publications as well as over 75 US patents to his credit. Vasudevan is a co-author of five books including two children’s books. At Makerville. Tech, he co-founded a start-up to create engaging books and educational resources that make AI and related technology accessible to young learners, parents, and educators. Dr. Vasudevan also serves as a Technical Advisor to start-ups and SMB’s providing guidance  on AI Strategy and applications. He can be contacted @linkedin.com/in/cvasudevankartha.

References