Skip to content
TheAgent Ecosystem
Automation

AI Vendor Due Diligence: Questions Worth Asking First

Capability you can measure yourself in an afternoon. Everything that binds you for years, you can only be told.

Muhammad Qasim HammadAI-assisted6 min read1,220 words

AI-drafted, reviewed by Muhammad Qasim Hammad on November 1, 2026. See our AI disclosure.

Procurement: Test the demo, read the contract
Table of contents
  1. Why not lead with capability?
  2. What should you ask about liability?
  3. What should you ask about data?
  4. What should you ask about change?
  5. How many of these actually get answered?

Most AI vendor evaluations test the demo and skip the contract. The AI vendor due diligence questions that matter are the ones whose answers you cannot discover later, and almost none of them are about model quality. Quality you can measure yourself in an afternoon. Everything below, you can only be told.

Why not lead with capability?#

Because capability is the only thing you can establish independently. Run 200 of your own inputs through the thing and you will know more about its quality than any vendor questionnaire will tell you, and you will know it in a form that matches your actual traffic rather than a benchmark.

Everything else is asymmetric. What happens to your data, what the liability cap is, how deprecation is handled and what provenance the output carries are all facts held by the vendor, invisible from the outside, and expensive to discover through experience. That asymmetry is what due diligence is for.

A useful ordering rule follows from that. Spend 1 afternoon measuring quality on your own inputs and 6 emails on the things you cannot measure, rather than the reverse. The second list is shorter, duller, and decides more.

Teams get this backwards because capability is interesting and contracts are not. The result is a thorough capability assessment attached to a signature on terms nobody read, and the terms are the part that binds for the next 3 years.

Comparison of facts a buyer can establish independently against facts only the vendor holdsDue diligence exists for the right column. The left column is a task, not a question.

What should you ask about liability?#

The cap, and how it compares to what you have promised your own customers. A cap set at some months of fees is normal and is unrelated to the size of harm an autonomous system can cause, so the gap between the two numbers is your exposure whether or not anybody has priced it.

Ask specifically what is excluded, because exclusions do more work than the cap. Output accuracy is commonly excluded entirely, which means the vendor warrants that the service runs rather than that it is right, and that distinction is the whole substance of what you are buying.

QuestionCheap to ask nowExpensive to discover later
What is your liability cap?One emailAfter a customer claim
What is excluded from warranty?One emailWhen output is wrong at scale
Where is data processed and retained?One emailDuring a compliance review
What provenance does output carry?One emailOn a labelling deadline
What is the deprecation notice period?One emailWith 30 days to migrate
What happens if a regulator restricts the model?One emailOn the day it happens
Table of six vendor questions with the cost of asking now against the cost of discovering the answer laterEvery row costs one email today. The right column is what the same answer costs when it arrives uninvited.

The last row is new and most contracts have nothing to say about it. Regulators can now require a provider to take corrective measures including restricting a model's public availability, which is a supply event arriving on a regulatory timetable rather than a commercial one.

What should you ask about data?#

Where it is processed, how long it is retained, whether it trains anything, and how deletion works. Those four are standard and worth restating because the answers vary far more between vendors than the marketing does, and the second and fourth interact badly with your own retention obligations.

Deletion deserves a follow-up question rather than a yes. Ask what happens to data already inside logs, caches and evaluation sets, because a vendor can honestly answer that customer data is deleted while derived copies persist somewhere the deletion pipeline does not reach.

Checklist of six vendor answers that are technically true while leaving the important question unansweredEach of these is honest as stated. Each leaves the thing you actually asked about unresolved.

Training use has 2 separate answers and both matter: whether your inputs train the vendor's models, and whether they are retained for abuse monitoring by human reviewers. A vendor can truthfully say your data does not train anything while a person can still read it under a different policy.

Residency answers should name regions rather than describing intent. A commitment that data stays within a region is a different thing from a statement that it is usually processed there, and the difference only becomes visible when somebody asks you to prove it.

What should you ask about change?#

Deprecation notice, version pinning, and whether behaviour can change under a stable version name. The third is the one people forget, and it is the one that breaks systems quietly, because the endpoint keeps working and the outputs stop matching what your prompts were tuned against.

Four step vendor assessment sequence putting independent capability testing before contract questionsStep 1 is the only part you can do alone, which is exactly why it should not consume the whole evaluation.

Version pinning is worth paying for where it is offered. A pinned version that stays available for a stated window converts an unpredictable behavioural change into a scheduled migration, which is the difference between a planned week and an unplanned one.

Ask for the active parameter count too, if you are ever considering hosting anything yourself. A model can be openly licensed and still sit far beyond what you would put on a cluster, and the ratio between total and active parameters tells you which situation you are in before you plan around it.

How many of these actually get answered?#

Fewer than you would like, and the pattern of refusals is itself information. A vendor that answers the data and deprecation questions crisply and goes vague on liability is telling you where the risk sits, and that is a more useful signal than a completed security questionnaire.

Decision flowchart for AI vendor assessment, testing quality on your own inputs, liability gap and deprecation terms before signingA vendor that will not answer the liability question in writing has answered it.

Send them as a short list rather than a document. Six questions in an email get answered by a person who knows the answers; a 40-page assessment gets routed to someone whose job is to return it completed, and the second process produces paper rather than information. What you owe users when the system is wrong is the reason the liability answers matter more than the capability ones.

Frequently asked questions

What should we ask an AI vendor before signing?
The things you cannot discover yourself: the liability cap and what is excluded from warranty, where data is processed and retained and whether it trains anything, the deprecation notice period and whether versions can be pinned, what provenance generated output carries, and what happens if a regulator restricts the model's availability. Capability you can measure independently in an afternoon.
Why does the liability cap matter so much?
Because it is frequently unrelated to the harm an autonomous system can cause, and because your own customer commitments are set separately. If the liability you accept towards customers exceeds what your vendor accepts towards you, that gap sits with you by default. Ask about exclusions too, since output accuracy is commonly excluded entirely.
What is the question people forget about model changes?
Whether behaviour can change under a stable version name. Deprecation notices get attention because they are announced, but a model that shifts underneath an unchanged endpoint breaks systems quietly: calls keep succeeding while outputs stop matching what your prompts were tuned against. Ask whether version pinning is available and for how long.
Is a completed security questionnaire good enough?
Usually not, because of who fills it in. A 40-page assessment gets routed to someone whose job is to return it completed, which produces paper. Six specific questions in an email get answered by a person who knows the answers, and the pattern of what they answer crisply against what they go vague on is itself useful information.
What should we ask about deletion?
What happens to data already inside logs, caches and evaluation sets, rather than accepting a yes. A vendor can honestly state that customer data is deleted while derived copies persist somewhere the deletion pipeline does not reach, and that distinction only surfaces during a compliance review or a subject request.

Written by

Muhammad Qasim Hammad
Muhammad Qasim Hammad
AI agents & automationFounder · Cart Gaze LLCPMP-certified PM

Muhammad Qasim Hammad is an AI agent and automation expert and the founder of Cart Gaze LLC (cartgaze.com). He builds product for the love of it: when an idea lands, a working prototype is usually running within hours, built with the same AI agents and automations he sells. He puts his own output at roughly 20× what it was before agents, and the Agentic OS behind this site is the working proof, documented in public with the tools he actually ran and what they really cost.

AI & Automation Services

Want a pipeline like this running in your business?

I'm Qasim — I design and ship AI agents and n8n automations for solo operators and small teams. Tell me what's eating your team's week, and I'll scope a fix.

Related reading