Hi all,
A workflow can give the right answer for the wrong reason. Evaluations in Agent Mesh can now catch that.
A dataset example can carry an expected path: which workflow nodes should run, succeed, fail or be skipped, and which tools should or shouldn’t be called. It can also carry expected facts, values the response must contain, like an order number, and values it must not, like a card number. Two built-in evaluators grade them. Workflow Path works from the captured events with no LLM involved, and Output Facts ignores case, spacing and where a value sits in JSON.
An LLM judge can also grade a single node with {{Node Output: <node id>}} or {{Node Input: <node id>}}. In the Experiment Lab you can now set Max Workers and pick a judge’s model with Edit Model.
I walk through a run of an order workflow here:
Which node in your workflows would you want a judge looking at on its own? Let me know in the comments.
If you want to try it yourself, you can download Solace Agent Mesh. Check it out on Github: