Anerli wrote: “When the interface does eventually change, that's where it becomes non-deterministic again by necessity, since the planner will need to generatively update the test and continue building the new cache from there.”
But what determines that the UI has changed for a specific URL? Your software independent of the planner LLM or do you require the visual LLM to make a determination of change?
You should also stop saying 100% open source when test plan generation and execution depend on non-open source AI components. It just doesn’t make sense.
The small VLM (Moondream) decides when interface changes / its actions no longer line up.
We say 100% open source because all of our code (test runner and AI agents) is completely open source. It’s also completely possible to run an entire OSS stack because you can configure with an open source planner LLM, and Moondream is open source. You could run it all locally even if you have solid hardware.