Managing Variability and Ensuring Stability in LLM-Based AI Test Automation: Strategies for Controlling Non-Deterministic Behavior and Temperature Effects to Minimize Test Failures Caused by Dynamic Interpretation of YAML Step Definitions #20
Description
What is your opinion on the variability in test responses given that the system is entirely LLM-based and does not cache or store locators? There is a high likelihood that a test passing today could fail tomorrow because the LLM might fail to interpret a specific step in the YAML correctly. Essentially, how can one control or manage this inherent randomness or “temperature” in the results?