Key takeaways
✓Agentic tools act on your systems autonomously, so the questions you ask vendors must go well beyond features and pricing.
✓The most important things to establish upfront: what the agent can do without human approval, what data it can access, and what happens when it makes a mistake.
✓Vendors who cannot clearly answer questions about scope, failure handling, and audit trails are not ready for enterprise deployment.
✓Treat these 12 questions as a minimum bar, not a complete due diligence process. Your security and legal teams should still review any tool that touches sensitive systems.
✓Your people need to understand how agents work before they can govern them well. Procurement and IT staff who lack that foundation will struggle to spot weak vendor answers.
Why evaluating AI agents is harder than evaluating other software
Most enterprise software does what you tell it, when you tell it. You click a button, a report runs. You submit a form, a record is created. The system is passive until a human acts.
AI agents work differently. You give an agent a goal, and it decides how to pursue it. That might mean calling an external API, reading and writing files, sending messages, querying a database, and making branching decisions along the way, all without a human approving each step. That autonomy is the point. It is also what makes procurement genuinely difficult.
When you evaluate a conventional SaaS tool, you are assessing a defined set of features. The tool either has a capability or it does not. Failure is usually visible and contained: a field does not populate, a query times out, a report shows an error. You notice, you fix it, you move on.
With an agentic tool, failure can be invisible until it is expensive. An agent that misinterprets a goal does not stop and ask for clarification. It continues executing, step after step, with increasing confidence in a direction that was wrong from the start. By the time a human notices, the agent may have sent emails, modified records, or triggered downstream processes that are not easy to reverse.
The core procurement risk with AI agents
Because agents act across systems and over multiple steps, a single ambiguous instruction can propagate into real-world consequences before anyone reviews the output. Standard software QA is not designed to catch this class of failure.
There is also the question of autonomy level. Not all agents are equal. Some are tightly scoped, able to take one or two defined actions in one system, with a human in the loop for anything significant. Others operate across multiple platforms, manage their own subagents, and are designed to run unsupervised for hours or days. The evaluation criteria that apply to a simple document-drafting assistant are wholly inadequate for an agent that can execute financial transactions or modify production systems.
That gap between autonomy levels is one reason standard IT procurement checklists do not transfer cleanly to agentic tools. Most vendor questionnaires were written when software could not take actions. Updating them is not optional.
What does the agent actually do, and when does it stop?
This sounds obvious, but many procurement conversations skip straight to pricing and integrations before anyone has pinned down the agent's actual scope of action. An AI agent that "automates your procurement workflow" could mean anything from drafting a purchase order for a human to approve, to raising, routing, and finalising that order without any human in the loop. Those are not the same thing, and the gap between them carries real risk.
Before evaluating anything else, get precise answers to these three questions.
Question 1: What actions can this agent take in the real world?
Ask the vendor for a complete, specific list of the agent's executable actions: what systems it can write to, what it can send, what it can delete or modify, and what it can trigger downstream. Not capabilities in principle. Actual actions, in your environment.
A well-scoped agent should have a clear action boundary. If the vendor struggles to give you that list, or the answer keeps expanding as you ask follow-ups, treat that as a design problem rather than a feature.
Question 2: What stops the agent from acting?
Every agent needs explicit stopping conditions: the situations in which it pauses, escalates, or refuses rather than continues. Ask the vendor to walk you through what happens when the agent encounters an ambiguous instruction, a missing input, a transaction above a certain value, or data it has not seen before.
If the answer is essentially "the model uses its judgement", push harder. Judgement without defined limits is not a stopping condition. You want to see rule-based guardrails, not just the model's general tendency toward caution.
The question every procurement team should ask
"Show me a scenario where this agent stops and hands back to a human. Walk me through exactly what triggers that, and what the handoff looks like." If the vendor cannot demo this, the stopping logic is not production-ready.
Question 3: Where is the human in the loop, and is that configurable?
Human-in-the-loop (HITL) is the design pattern where a person reviews or approves the agent's output before it takes effect. It is the most practical safeguard available, and whether it is present, and how it is configured, is one of the most important things to establish early.
Ask specifically: is HITL on by default or off? Can you require approval for certain action types (high-value transactions, external communications, file deletions) without disabling automation everywhere else? And who receives the approval request when the agent escalates?
The four levels of AI agent autonomy framework is useful here. A fully autonomous agent operating at level four is appropriate in very few enterprise contexts today. Most teams benefit from starting at level two or three, where the agent prepares and recommends but a human confirms. Know which level you are buying before you sign anything.
How does the agent handle data and access?
This is where many procurement conversations go thin. Vendors demo the capability; they are less forthcoming about where your data goes, who can read it, and what the agent is allowed to touch.
Question 4: Where is data processed and stored, and does that meet your obligations?
For Australian organisations, this is not a preference, it is a compliance question. Under the Privacy Act and the Australian Privacy Principles, you need to know whether personal information is being sent offshore, to which jurisdictions, and under what safeguards. Some agentic platforms process inputs through US-based infrastructure by default. Others offer regional data residency, sometimes at extra cost. Ask for this in writing, not in a sales deck.
If you are in the public sector, the bar is higher again. The ASD Infosec Registered Assessors Program and your agency's data classification requirements may significantly constrain which platforms are permissible. Our article on AI in the Australian public sector covers the specific considerations there.
Question 5: What permissions does the agent request, and are they scoped to what it actually needs?
An agent that connects to your Microsoft 365 environment or your CRM does not need write access to every mailbox and every record to do its job. The principle of least privilege applies here just as it does with human users. Ask the vendor to show you, specifically, which permissions the agent requests during setup and why each one is necessary.
Watch for agents that request broad OAuth scopes ("read and write all files") when the actual workflow only requires access to a specific folder or data source. Overpermissioned agents are a significant risk, not because the vendor is malicious, but because every permission the agent holds is a permission an attacker can abuse if the agent is compromised or misconfigured. Our seven data leakage scenarios article covers how this plays out in practice.
Question 6: What can the agent read, write, modify, or delete, and is that list complete?
This question is different from the permissions question. Permissions are what the system allows. This question is about what the agent is designed to do with that access. Some vendors document this clearly. Others leave it implicit in the product behaviour.
Ask for a written description of every action type the agent can take: read, write, send, delete, create, approve, trigger. Then ask which of those actions are reversible. A mistake in a read-only agent means a wrong answer. A mistake in an agent that can delete records, send emails on your behalf, or approve transactions means a problem you may not be able to undo.
Reversibility matters more than you think
Most incident response plans assume a human made the error. When an agent acts at speed across many records or messages, the window to catch and reverse a mistake is much shorter. Know before you deploy which actions can be rolled back and which cannot.
This connects directly to how you think about agent autonomy levels. A fully autonomous agent that can both decide and act without human confirmation needs a much tighter permission boundary than an agent that only surfaces recommendations for a person to approve.
What happens when the agent gets it wrong?
Every agent will make mistakes. The question is whether the vendor has built the system assuming that, or assuming it won't happen. Three questions sort confident vendors from optimistic ones.
Question 7: How does the agent signal uncertainty or failure?
Some agents fail silently. They complete a task, return an output, and give you no indication that something went sideways. Others are built to surface low-confidence decisions, flag ambiguous inputs, or pause and request human review before proceeding. Ask the vendor to show you both: a demo of normal operation and a demo of what failure looks like. If they cannot show you the second one, that tells you something.
Also ask whether failure handling is configurable. A finance team approving vendor payments needs different thresholds than a marketing team drafting social posts. The agent's tolerance for uncertainty should match the stakes of the task it is running.
Question 8: What audit trail does the agent produce?
When an agent takes an action, you need a record of what it decided, why, and what it did. This matters for incident response, for compliance, and for the basic operational question of "how did we end up here?" Ask specifically whether the log captures the agent's reasoning steps, not just the final output. Some vendors log inputs and outputs only. That is often not enough.
For Australian organisations operating under the Privacy Act or sector-specific obligations, the ability to reconstruct an agent's decision path is not optional. If the vendor's answer to this question is vague, escalate it. A well-designed audit trail is also what makes verification protocols for high-stakes requests practicable rather than theoretical.
Question 9: Can the agent's actions be rolled back, and how?
Some actions are reversible. A drafted email that was never sent, a document flagged for review rather than deleted. Others are not: a payment submitted, a record permanently overwritten, an external API call completed. Ask the vendor to map out, for your specific use case, which actions the agent takes are reversible and what the rollback process looks like.
This is closely related to the autonomy question. As we covered in the four levels of AI agent autonomy, higher-autonomy agents take more irreversible actions, which raises the cost of errors proportionally. If the vendor cannot give you a clear answer on rollback, treat that as a signal to constrain the agent's scope before deployment, not after something goes wrong.
The cost of an error scales with autonomy
The more independently an agent acts, the more expensive its mistakes become. Audit trails and rollback capabilities are not edge-case features. They are what make high-autonomy deployment defensible.
How does the vendor handle change and support?
Most procurement checklists stop after security and accuracy. That is a mistake. Agentic tools are unusually exposed to change, because the underlying model is rarely static. Vendors update it, sometimes silently, and an agent that worked reliably in March can behave differently in June with no action on your part.
These three questions are about what happens after you sign.
Question 10: How and when does the vendor notify you of model updates, and what is the testing period before changes go live?
For a conventional SaaS tool, a UI refresh is an inconvenience. For an agentic workflow, a model update can alter how the agent interprets instructions, which data it surfaces, or whether it completes a task at all. Ask vendors to show you their change-management process in writing. Specifically: do updates roll out automatically to production, or is there a staging environment where you can test first? If the answer is "automatic, with an email notification," that is worth flagging as a risk before you build anything critical on top of their platform.
Question 11: Does the vendor offer workflow versioning, and can you pin to a specific model version?
Some vendors let you lock a workflow to a particular model version so that updates do not touch it until you choose to migrate. Others do not. If the agent is embedded in a high-stakes process, such as contract review or financial approval, version pinning is not a nice-to-have. It is the thing that stops a background model update from quietly changing an output that no one notices until an audit.
The silent update problem
Agents can fail in ways that look like success. A model update might cause an agent to skip a step, reinterpret a field, or change its output format slightly. If no human is checking the output, you may not discover the regression for weeks.
This connects directly to how you design human oversight into agentic workflows. The higher the autonomy level, the more damage a silent model change can do.
Question 12: What does support actually cover, and who owns the agent when something breaks?
This question exposes a gap that catches a lot of enterprise buyers off guard. Many agentic platforms are sold with standard SaaS support: ticket queue, knowledge base, SLA for uptime. But agentic failures are often ambiguous. The platform is up, the model is responding, and yet the agent is producing wrong outputs because a prompt template interacted unexpectedly with a model update. Whose problem is that?
Ask the vendor to describe a real support scenario involving a regression after a model update. How do they triage it? Who is responsible for restoring the workflow? If the answer is vague, or if it places most of the diagnostic burden on your team, factor that into your internal resourcing plan. You will need someone who understands both the tool and your workflows well enough to debug the gap between them. That capability sits with your people, not the vendor.
How to use these questions in a real procurement process
The twelve questions above are most useful when you assign them to a specific stage. Dumping all of them into an initial vendor conversation wastes everyone's time and signals that you have not done the work to prioritise. Structure the evaluation in three phases and the process gets sharper at every step.
RFI: filter before you engage
An RFI (Request for Information) is a written questionnaire you send before any sales call. Keep it short. Four to five questions is enough to separate vendors worth talking to from those who will consume your calendar without a credible answer.
Good RFI questions from the list above:
What actions can the agent take without human approval?
Where does data go, and does it leave Australian jurisdiction?
What audit logs are available, and who can access them?
Any vendor that cannot answer these in writing is telling you something. Vague or marketing-heavy responses at this stage are a reliable indicator of vague answers under pressure later.
Demo: watch, do not just listen
A demo is where vendors are at their most persuasive, and where buyers are most likely to get swept along. Go in with a prepared scenario that reflects your actual use case, not the one the vendor has rehearsed. Ask them to run the agent through a task that involves a judgment call or an ambiguous input.
Specifically, ask them to show you what happens when the agent fails. What does the error look like? Does it halt and notify, or does it continue quietly? Vendors who cannot demonstrate a failure gracefully have either not tested for it or are hoping you will not ask.
This is also the right moment to ask about the human-in-the-loop controls you read about in the autonomy levels discussion. If you have already worked through the four levels of AI agent autonomy, you will know exactly which level you are comfortable deploying, and you can ask the vendor to show you that configuration directly.
Proof of concept: run a bounded real task
A proof of concept (PoC) is a short, time-limited deployment on a real but low-risk workflow. Two to four weeks is usually enough. The point is not to evaluate the full product. The point is to see how the agent behaves on your data, in your environment, with your edge cases.
Define success criteria before you start, not after. Typical criteria for an agent PoC might include:
Task completion rate on a defined set of inputs
Number of escalations or human interventions triggered
Audit log completeness and readability
Time taken for your IT team to configure and monitor the deployment
At the end of the PoC, the answers you got in writing during the RFI should match what you observed in practice. When they do not, that gap is the most important finding of the entire evaluation.
The PoC reveals what the RFI cannot
A vendor's written answers describe the product as designed. A proof of concept shows you how it behaves under real conditions. Both matter, and neither replaces the other.
One more thing worth building into your process: involve the people who will actually use the agent. IT and procurement can assess security and contractual terms, but the team doing the underlying work will catch friction points that no procurement checklist will surface. Getting that input before sign-off, rather than after rollout, is where workflow design comes before tool selection in practice.
Frequently asked questions
What is the difference between evaluating a standard AI tool and evaluating an AI agent?
A standard AI tool responds to a prompt and stops. An agent takes actions, often across multiple systems, without a human approving each step. That changes the risk profile entirely. When you evaluate a conventional tool you are largely assessing output quality. When you are evaluating AI agents you are assessing decision logic, access controls, failure handling, and audit trails. The questions are fundamentally different, and the procurement process needs to reflect that.
How do we know if a vendor's safety claims are genuine?
Ask for evidence, not assurances. A vendor who says their agent "never accesses data it shouldn't" should be able to show you the permission model in writing, demonstrate it in a sandbox, and point to independent audit results or penetration test summaries. If they respond with marketing language instead of technical documentation, treat that as a red flag. You can also ask your IT team to run a limited proof-of-concept in an isolated environment before committing to a full deployment.
How many of these questions should we expect a vendor to answer on first contact?
A credible vendor should be able to answer the majority of them, at least at a high level, in an initial scoping conversation. Questions about data residency, permission scoping, and rollback are standard due diligence for any enterprise software purchase. If a vendor is evasive or defers everything to a later "technical deep-dive" that never quite arrives, that tells you something about how they will handle support issues after you sign.
Should IT lead this evaluation, or should it involve other parts of the business?
IT should own the security and infrastructure questions, but they should not run the evaluation alone. The team that will actually work alongside the agent understands the edge cases, the exceptions, and the moments where human judgement matters most. Procurement brings contract and vendor risk expertise. And someone with governance or compliance responsibility needs to be involved from the start, not brought in at the end to rubber-stamp a decision already made. For a practical way to structure those roles, see the five roles every enterprise AI initiative actually needs.
Is there a point at which an agent is simply too autonomous to deploy safely?
Yes. If an agent can take irreversible actions at scale without a human checkpoint, and the vendor cannot clearly explain how that behaviour is constrained, that is a deployment risk most organisations are not equipped to absorb. The four levels of AI agent autonomy framework is useful here: it helps you name what level of autonomy a given tool actually operates at, and match that to what your organisation is genuinely ready for. Autonomous does not mean unsafe by definition, but it does mean the controls need to be proportionately stronger.
Ready to build the skills to run this evaluation well?
Asking the right questions is half the work. The other half is having people in your organisation who understand the answers well enough to act on them.
Evaluating AI agents sits at the intersection of IT architecture, data governance, and operational risk. Most procurement teams are strong on one of those three. Few are strong on all three. That gap is where vendor answers go unchallenged and where commitments that looked solid on paper quietly unravel after go-live.
Better People works with enterprise and government teams to build exactly that internal capability: the practical understanding of how agents work, what can go wrong, and what good governance looks like before, during, and after deployment. If your team is about to evaluate agentic tools for the first time, or if a previous rollout taught you things you'd rather not repeat, a structured conversation about where the gaps are is worth having early.
Is your team ready to evaluate agentic AI tools with confidence?
We'll talk through your procurement context, where the technical and governance gaps tend to appear, and what practical preparation looks like for your team.
Book a 30-minute discovery call →
You can also explore our AI implementation services if you're looking for structured support across the full evaluation-to-deployment cycle, not just the vendor question stage.
