How to evaluate OpenAI and other AI vendors

A 20-question checklist for OpenAI and other AI vendors covering roles, data, retention, sub-processors, security, change and exit.

Publicat 1 august 20265 minutes

AI ACT · TECHNOLOGY PROCUREMENT

An AI vendor assessment is not a collection of security badges. It must connect the purchased product to your use case, submitted data, legal role, technical chain and the real ability to change or stop the service.

Start with product and role, not the company name

The same vendor may offer consumer applications, business workspaces, APIs, hosted models and connectors under different terms. Assess the configuration you will use. Then determine whether your organisation is a system deployer, a provider under its own name or has another role; integrating an upstream model does not create one automatic classification. ^[EU-REG] ^[EU-GPAI-G] ^[EU-DESK]

The first four questions define what is being bought. Without them, answers on retention, location or security may be correct for another product and useless for your project. ^[EU-GPAI-G] ^[OAI-DPA]

1. Which exact product, model, version, region and features are we buying?
2. What is the purpose, who uses it and which decision or action does it influence?
3. What AI Act role does each party have for this specific use case?
4. What documentation supports classification and downstream integration?

Data, training, retention and location

OpenAI says it does not, by default, train on business-product and API inputs or outputs unless the organisation explicitly opts in. This is important but must be tied to product, account and settings. Separate training use from operational retention, moderation, support, security and your own logs. ^[OAI-DATA] ^[OAI-DPA]

OpenAI’s DPA makes the customer responsible for certain settings, including retention and deletion, and describes transfers for EEA data. Do not accept “data stays in Europe” without checking the product, selected region, involved entities, exceptions and contractual mechanism. ^[OAI-DPA]

5. What data enters, leaves and is retained at each stage?
6. Are inputs or outputs used for training, and under which setting?
7. What are the retention periods, exceptions and deletion options?
8. Where is data processed and which mechanism covers transfers?

Sub-processors, security and incidents

OpenAI’s 9 July 2026 list distinguishes sub-processors by product, purpose and location, including entries conditional on customer choice. The DPA provides change notice, a 30-day objection right and comparable obligations for sub-processors. Retain the reviewed version and name the owner monitoring changes. ^[OAI-SUB] ^[OAI-DPA]

Do not assess security through certifications alone. Request controls relevant to the flow: identity, tenant segregation, encryption, key management, administrative access, log export, vulnerabilities, incident notice and investigation support. Evidence should match the risk of the data and actions permitted to the system. ^[OAI-DPA] ^[EU-REG]

9. Which sub-processors touch data, for what purpose and where?
10. How are we notified and what can we do when the chain changes?
11. Which security controls apply to the purchased configuration?
12. How quickly and with what information are incidents reported?

Quality, transparency and model change

Model documentation and signing a code of practice are useful signals, not substitutes for use-case testing. OpenAI announced signing the GPAI Code and the Commission lists it as a signatory. Still verify the model, published limits, downstream information and whether prompts, data, tools or actions in your system change the risk profile. ^[OAI-ACT] ^[EU-GPAI] ^[EU-GPAI-G]

Define an evaluation set before purchase: correct answer, correct refusal, source citation, multilingual behaviour, attacks, sensitive data, tool failure and cost. Retain results per version. A moving model alias can alter quality without code changes, so contract and monitoring should treat version as a dependency. ^[EU-GPAI-G] ^[EU-LIT]

13. What limitations, evaluations and documentation are published for the model?
14. How will our interface meet user-facing transparency duties?
15. How are versions, deprecations and material changes communicated?
16. Which tests must each model pass before production promotion?

Cost, control and the exit plan

A suitable vendor today may become unsuitable through price, limits, region, terms, performance or strategy. The contract should define exportable data, format, post-termination access and deletion of copies. Technically, separate product logic from the vendor API and document degraded operation without the model. ^[OAI-DPA] ^[EU-REG]

A final score must not hide a disqualifying criterion. A vendor may score well overall yet remain incompatible with sensitive data, required residency or an irreversible action. Record risk acceptance with an owner and deadline, not inside an average. Reassess on material change and at the interval set in the AI inventory. ^[EU-DESK] ^[EU-REG] ^[OAI-SUB]

17. What is total cost at scale, including logs, evaluation and support?
18. Can permissions, budget and actions be limited by team and environment?
19. Which data, prompts, evaluations and configurations can we export on exit?
20. Who decides reassessment, suspension and replacement of the vendor?