Key takeaways
✓Government AI procurement sits under existing Commonwealth and state frameworks, but standard ICT criteria miss several AI-specific risks that need separate assessment before a contract is signed.
✓Risk categorisation comes first. An AI tool used for low-stakes internal drafting sits in a different risk tier than one influencing citizen-facing decisions, and the due diligence for each is different.
✓Data sovereignty is not optional. Where model training data goes, where inference happens, and who can access agency data under the vendor's terms of service all require written answers before evaluation.
✓Vendor capability claims need verification. Many products marketed as "AI" are narrow automation tools; understanding what you are actually buying changes both the risk profile and the build-versus-buy calculus.
✓Staff capability is part of the procurement decision. Buying an AI system your team cannot evaluate, prompt, or audit creates dependency risk that no contract clause fully resolves.
Why is government AI procurement different from standard ICT purchasing?
Buying a new HR platform or cloud storage service is complicated enough. Government AI procurement adds layers that standard ICT purchasing does not carry, and underestimating that difference is where agencies run into trouble.
The core issue is accountability. When a government agency uses AI to assist with a decision, such as assessing eligibility for a benefit or flagging a procurement anomaly, the agency remains legally responsible for that decision. The AI vendor does not. That asymmetry shapes everything from how you write the contract to how you document the system's behaviour after go-live.
Standard software procurement asks: does it do what it claims, is it secure, and is the price reasonable? AI procurement asks all of those, plus several harder questions.
Explainability. Can the system tell you, in plain terms, why it produced a particular output? Many AI models cannot, and that matters when a minister or tribunal asks for the reasoning behind an automated decision.
Bias and fairness. AI systems trained on historical data can embed historical patterns of disadvantage. Government agencies serving the whole population carry a higher duty of care here than a private business does.
Ongoing drift. A traditional software package behaves the same on day one and day three hundred. AI models can change as they are updated or as the underlying data shifts. Your procurement needs to account for how the vendor handles model versioning and how you will detect when performance degrades.
Auditability. Public sector decisions are subject to freedom of information requests, parliamentary scrutiny, and audit office review. The audit trail an AI system produces needs to be readable by people, not just machines.
There is also the policy layer. Australian government agencies operate under the Australian Government Voluntary AI Safety Standard, the Privacy Act 1988, and, depending on jurisdiction, a range of state and territory frameworks. Some agencies also operate under security classifications that limit which cloud environments are permissible and which data can leave Australian borders.
The accountability gap
AI vendors sell capability. Government agencies carry responsibility. That gap does not close itself, and a contract that does not address it explicitly will leave the agency exposed.
Procurement teams who have only worked on standard ICT deals often treat AI as a software subcategory. Legally and operationally, it is closer to a decision-support system with a probabilistic output, which is a different thing entirely. Understanding that distinction early is what separates agencies that deploy AI confidently from those that stall at the pilot stage.
For a fuller picture of what is and is not currently permissible under federal and state policy, see AI in the Australian public sector: what's allowed.
What frameworks govern AI procurement in Australia?
No single Australian law governs AI procurement end-to-end. Instead, agencies work across a layered set of policies, frameworks and guidance documents, each covering a different dimension of the purchase.
The Australian Government's AI Ethics Principles are the starting point. Published by the Department of Industry, Science and Resources, the eight principles (including human oversight, fairness, privacy protection and transparency) are not legally binding on vendors, but Commonwealth procurement documentation increasingly asks suppliers to demonstrate alignment with them. Agencies that ignore the principles during evaluation risk approving tools that contradict official policy positions down the track.
The Digital Transformation Agency (DTA) sets the overarching ICT procurement rules for Commonwealth entities through the ICT Procurement Framework and the Digital Sourcing Policies. These require agencies to assess value for money, consider Australian industry participation, and document risk. The DTA has also published AI-specific guidance (under its "Digital and ICT Investment" work) that encourages agencies to apply proportionate scrutiny to AI tools, particularly those that make or influence decisions affecting citizens. The DTA guidance does not carry the force of a regulation, but it informs how the Australian National Audit Office (ANAO) will assess a procurement decision if it is ever reviewed.
The ASD Essential Eight is a different animal. Developed by the Australian Signals Directorate, it is a set of eight mitigation strategies designed to reduce cyber security risk. It applies to Commonwealth systems and, by extension, to any cloud or AI platform that connects to them. Procurement teams often treat the Essential Eight as an IT security checklist, but its relevance to AI procurement is broader: an AI tool that requires elevated privileges, handles sensitive datasets, or calls external APIs may introduce exposure across multiple Essential Eight controls simultaneously.
The Essential Eight is not just an IT concern
An AI platform that ingests citizen data, calls external services, or uses a large language model hosted offshore can create exposure across patch management, application control, and multi-factor authentication requirements at the same time. Security and procurement teams need to assess these together, not in sequence.
State and territory frameworks vary considerably. The NSW Government has a Artificial Intelligence Strategy and associated assurance framework. Victoria has published AI guidelines through the Department of Government Services. Queensland and Western Australia have issued guidance notes through their respective ICT procurement bodies. The level of prescription differs, but the common thread is that all of them ask agencies to assess AI risk before deployment, not after. If your agency operates under state rather than Commonwealth rules, it is worth pulling the specific guidance from your state's digital or services department, because the Commonwealth DTA framework does not automatically apply.
The Privacy Act 1988 and the Australian Privacy Principles (APPs) add another layer. AI tools that process personal information held by an agency must be assessed against the APPs, particularly around data use, disclosure to third parties, and storage offshore. This intersects directly with sovereignty concerns covered in the next section, and with what Australian agencies are currently permitted to do with AI tools.
Taken together, these frameworks mean government procurement teams are simultaneously accountable to ethics policy, procurement policy, cyber security standards, and privacy law. No vendor assessment process that addresses only one of those dimensions will survive scrutiny.
How should agencies categorise AI risk before buying?
Risk categorisation should happen before a vendor is shortlisted, not after. The use case and the data it touches determine the risk level, and the risk level determines how much scrutiny the procurement needs. Getting this sequence right saves significant time later.
A practical starting point is to ask two questions about every proposed AI tool. First: what decisions will this tool inform or automate? Second: what data will it process to do that? The answers place the tool in one of three broad tiers.
Tier 1: Low risk
Tools in this tier support administrative tasks where no sensitive data is involved and no consequential decision rests on the output. Examples include drafting internal communications, summarising publicly available documents, or generating meeting notes from a pre-approved transcript. Errors are catchable and reversible. Procurement can follow a lighter pathway, though probity and supplier checks still apply.
Tier 2: Medium risk
These tools process internal agency data or inform decisions that affect staff. A document management system with AI-assisted classification, or a procurement tool that ranks suppliers using agency spend data, sits here. The consequences of a wrong output are meaningful but contained. Agencies should require documented model explainability, clear human-review checkpoints, and confirmation of where data is stored and processed.
Tier 3: High risk
Any AI system that processes personal information, informs decisions affecting citizens, or touches classified or protected data belongs here. Automated benefits assessments, identity verification tools, and any system connected to law enforcement or national security data are obvious examples. Procurement at this tier requires a formal privacy impact assessment, a detailed security risk assessment aligned to the Australian Government's Information Security Manual (ISM), and ministerial or senior executive sign-off depending on agency policy.
The use case drives the tier, not the tool name
A general-purpose AI assistant can sit in Tier 1 for one team and Tier 3 for another, depending on what data they feed it. Evaluate the deployment context, not just the product.
It is worth being explicit that these tiers are not static. A tool procured for low-risk drafting tasks can migrate to higher risk if scope creep occurs and staff begin feeding it sensitive casework documents. Procurement decisions should include conditions on permitted use cases, with a review trigger if the deployment expands. That condition belongs in the contract, not just in internal guidance.
For agencies that have not yet built a formal AI risk framework, the Department of Finance's guidance on AI use in government procurement and the National AI Centre's responsible AI resources are reasonable starting points. The Australian Signals Directorate's ISM provides the security baseline for higher-tier tools. Better People's article on AI in the Australian public sector covers what is currently permissible under Commonwealth and state policy, which feeds directly into how agencies should set their tier boundaries.
One practical recommendation: build the risk categorisation step into your procurement templates as a mandatory field, not an optional checklist. When the person raising a purchase request must nominate a risk tier and justify it, the agency gets early visibility over what is coming through the pipeline. That visibility is worth more than any retrospective audit.
What data sovereignty questions must be answered first?
Before any risk category is confirmed or contract drafted, agencies need clear answers on where their data actually goes. A vendor's marketing materials will say "hosted in Australia" while the terms of service quietly permit processing in Singapore or the United States. Those two things can both be true at once, and the difference matters enormously for OFFICIAL: Sensitive and PROTECTED workloads.
Start with three questions and do not accept vague answers to any of them.
Where is data stored at rest? The primary data centre location should be named explicitly, not described as "Asia-Pacific" or "sovereign-aligned." For data classified at OFFICIAL: Sensitive or above, storage must be within Australian borders unless a specific exemption applies. Ask for the physical facility name, not just the cloud region label.
Where is data processed? Storage location and processing location are not always the same thing. Inference requests, model fine-tuning, and logging pipelines may route through infrastructure in a different jurisdiction even when the data store sits in Australia. Get written confirmation of every location where any processing occurs, including temporary or cached states.
Who are the sub-processors, and where do they operate? Most enterprise AI vendors rely on a chain of third-party services: cloud providers, content moderation services, monitoring tools, and sometimes human review queues for model improvement. Each of those sub-processors represents a separate jurisdictional question. Ask for the full sub-processor list, not just the primary cloud provider, and check it against the data handling requirements in the Australian Government's ISM and any relevant Privacy Act obligations.
The sovereignty question that gets missed
Most agencies check where data is stored. Fewer check where it is processed, and fewer still ask who the sub-processors are and in which countries they operate. All three questions need written answers before a contract is signed.
For PROTECTED workloads, the bar is higher again. ASD-certified cloud services listed on the Hosting Certification Framework are the appropriate starting point for that classification level. If a vendor's product is not on that list and the intended use involves PROTECTED data, that is not a procurement question to resolve later in the process.
It is also worth distinguishing between a vendor's consumer-grade product and their government or enterprise offering. Several major AI platforms offer Australian data residency only on specific tiers, with the standard plan defaulting to global routing. Confirm which product tier is actually being procured and that the data handling commitments in the contract match that tier's documented behaviour, not the enterprise tier's marketing copy.
For a deeper treatment of where Australian government data sovereignty requirements come from and how they interact with cloud AI platforms, Data sovereignty and AI for Australian government covers the regulatory context in detail.
Which vendor due diligence steps are non-negotiable?
Once you have categorised the risk of a proposed AI system, vendor due diligence is where that risk assessment becomes a list of real questions with required answers. Many agencies treat this stage too lightly, accepting marketing language about "enterprise-grade security" and "responsible AI" without pressing for specifics. The following checks are not optional for Australian government procurement.
Model transparency
Ask the vendor to explain, in plain terms, how the model makes its decisions. You do not need a mathematics lecture, but you do need to know whether explanations can be produced for individual outputs. For higher-risk applications, for instance, anything that influences eligibility decisions or resource allocation, a system that cannot be interrogated is not fit for purpose. Ask specifically whether the model is a third-party foundation model (such as GPT-4 or Gemini), a fine-tuned version of one, or a proprietary build, because the answer changes your rights around audits and your exposure to changes the underlying vendor makes without notice.
Data retention and handling
Confirm exactly where agency data goes when it enters the system, how long it is retained, and who can access it. This is not a box-ticking step. Some AI platforms train on user inputs by default unless you opt out, and the opt-out process is not always straightforward. You need written confirmation of the retention period, the deletion process, and whether data is used for model improvement. For PROTECTED or higher classified data, check that the platform holds the relevant Australian Signals Directorate (ASD) certifications and that the data never transits foreign infrastructure. The data sovereignty considerations covered in the sibling article on this topic are a useful cross-reference here.
Auditability and logging
The system should produce logs detailed enough for a post-incident review. That means recording what inputs were submitted, what outputs were returned, and which version of the model was running at the time. Ask the vendor who holds those logs, how long they are kept, and whether your agency can export them independently. If the answer is that logs are held exclusively by the vendor and not accessible to you, that is a significant governance gap.
Human oversight mechanisms
Confirm that the system is designed to support, not replace, human decision-making wherever the output has legal or material consequences. This is consistent with the Australian Government's AI Ethics Principles, which require that humans remain accountable for decisions affecting individuals. In practice, ask how the interface flags low-confidence outputs, whether there is a mandatory review step before outputs are acted on, and how overrides are recorded.
The due diligence question most agencies skip
Ask the vendor what happens when the underlying model changes. Foundation models are updated regularly, sometimes in ways that alter behaviour meaningfully. Your contract should require notification of material model changes and give you the right to re-evaluate before those changes take effect in your environment.
Contractual controls
The due diligence process only has teeth if its findings are reflected in the contract. At minimum, the agreement should specify the data classification levels the platform is approved to handle, the vendor's obligations on incident notification, your audit rights, and termination conditions if the vendor fails to meet them. Standard click-through SaaS agreements almost never cover these points adequately. Government legal and procurement teams should treat the vendor's standard contract as a starting position, not a final document.
If the vendor is unwilling to negotiate on data handling, audit rights, or model change notification, that tells you something useful before you sign.
How does AI capability fit into the procurement process?
Procurement gets you a contract. It does not get you adoption.
This is where many agencies stall. The tender is awarded, the system goes live, and then staff either avoid the tool or use it in ways the vendor never intended and the agency never evaluated. Neither outcome is acceptable when the tool handles citizen data or informs policy decisions.
Internal AI capability needs to develop in parallel with procurement, not after it. There are three points in the cycle where it matters most.
Before the tender. The people writing requirements and evaluating bids need enough AI literacy to ask hard questions. If your evaluation panel cannot distinguish between a rule-based automation and a large language model, they cannot assess whether a vendor's claims about accuracy, explainability or bias mitigation are plausible. A working understanding of how AI agents operate and where they fail is now a basic requirement for anyone involved in assessing an AI solution, not an optional extra.
During implementation. Workflow design should drive configuration choices, not the other way around. Agencies that let a vendor configure the system to its own defaults, then retrofit their processes around the output, consistently report worse outcomes than those who design the workflow before touching the tool. Subject matter experts from inside the agency need to be active participants in that design process, which means they need enough context to push back when a vendor's suggested workflow does not match operational reality.
After go-live. Ongoing AI literacy determines whether the system improves over time or drifts. Staff who understand how prompts affect outputs, how to recognise a hallucination, and when to escalate an anomaly are a quality-control layer that no contract can fully substitute for. That is not a technology problem; it is a capability problem.
Capability gaps compound over time
An agency that procures an AI system without building internal literacy will depend on the vendor to interpret every performance question, flag every risk, and justify every change. That dependency is a governance risk, not just an operational inconvenience.
The sequencing matters too. Teams that receive training before a system launches are measurably faster to adopt it and more likely to surface problems early. Waiting until staff complain is waiting too long. A structured approach to building AI literacy across the organisation should appear in the implementation plan, ideally with defined timelines and accountable owners, before the contract is signed.
For procurement officers specifically, the capability requirement is twofold: understand enough about AI systems to evaluate them fairly, and understand enough about your agency's workflows to specify them clearly. Neither skill is innate. Both can be built.
Better People works with Australian government agencies on exactly this problem, from pre-procurement literacy for evaluation panels through to post-implementation capability programs for frontline teams. The government training page outlines what that looks like in practice.
Is your team ready to evaluate and adopt AI tools effectively?
We work with Australian government agencies to build AI capability at every stage of the procurement cycle, from writing better requirements through to confident post-go-live adoption. Find out what a program for your agency could look like.
Explore government training options →
Frequently asked questions
Does AI software require a separate procurement process from standard software?
Not always, but it often should. AI systems that process personal information, influence decisions affecting individuals, or connect to external data sources carry risks that standard software assessments rarely capture. The Australian Government's AI Ethics Framework asks agencies to consider fairness, accountability, and explainability before deployment, and those questions sit outside a typical ICT risk checklist. If your agency is buying a tool that automates or informs a decision with real consequences for citizens, treat it as a distinct procurement category regardless of how the vendor classifies the product.
Can a government agency use a cloud-based AI tool if the data stays onshore?
Data residency is a necessary condition, not a sufficient one. Even when data is stored on Australian soil, it may still be accessible to offshore staff, processed by models trained on non-Australian data, or subject to foreign laws that compel disclosure. Agencies handling Protected or above information need to verify not just where data sits, but who can access it, under what legal jurisdiction the vendor operates, and whether the model's inference engine runs locally or phones home. The data sovereignty questions are worth working through methodically before any contract is signed.
What happens if an AI vendor cannot explain how their model makes decisions?
That is a red flag for high-risk use cases and a procurement concern worth escalating. The AI Ethics Framework's transparency principle requires that agencies be able to explain to affected individuals why a decision was made. If the vendor cannot provide that explanation, your agency inherits the accountability gap. For low-risk administrative tools, limited explainability may be acceptable. For anything touching eligibility, assessment, or enforcement, insist on documented model logic, audit trails, and a clear escalation path when the system produces an unexpected result.
Is there a standard government panel or contract vehicle for AI products in Australia?
The Digital Marketplace (now part of the ICT Procurement Panel managed by the Digital Transformation Agency) includes AI products and services, and many agencies use it as a starting point. State governments maintain their own panels, and some categories of AI have been added to existing whole-of-government arrangements. Using a panel simplifies probity but does not remove the obligation to conduct your own risk and fit-for-purpose assessment. A tool that cleared a panel evaluation two years ago may not reflect the capabilities or risks of the product on offer today.
How should agencies handle AI capability gaps when procuring a new system?
Buying the tool and building the capability need to happen on roughly the same timeline. Agencies that procure AI without investing in staff readiness tend to see low adoption, shadow workarounds, and governance failures when the system is used in ways its risk assessment never anticipated. The most effective approach is to include a workforce capability component in the business case from the start, covering not just technical users but the procurement officers, legal advisers, and operational staff who will interact with the system. What that training looks like in practice depends on the tool and the team, but building an AI literacy baseline across the agency is a reasonable place to begin.
Ready to build AI capability across your agency?
Procurement gets you the tools. What determines whether those tools actually change how your agency works is whether your people understand them well enough to use them well, question them appropriately, and spot the failure modes before they become problems.
That gap between signing a contract and realising value is where most government AI initiatives stall.
Better People works with Australian government agencies to build the practical AI capability that sits alongside procurement: helping procurement teams understand what they are evaluating, helping delivery teams work confidently with AI tools once they arrive, and helping executives ask the right questions throughout.
Is your agency ready to use the AI tools you're procuring?
We work with federal and state agencies on structured AI capability programs, from foundational literacy through to role-specific workflows. A short conversation is enough to map where the gaps are and what a practical program would look like.
See how we work with government →
If you are earlier in the process, the article on AI in the Australian public sector covers what is currently permitted under APS policy, which is a useful starting point before any vendor conversations begin.
