In 2009, a surgeon named Atul Gawande and a team backed by the World Health Organization showed that a 19-item surgical checklist could cut complications and deaths by dramatic margins across eight hospitals worldwide. Not a thousand-page protocol. Not a comprehensive framework. Nineteen items, printed on a single card. Aviation learned the same lesson decades earlier: the pre-flight checklist fits in a pilot’s hand, not in a binder. Nearly two decades later, I watch security teams send AI vendors questionnaires with 300 questions, half of which begin with “describe your approach to…” and almost none of which would catch a real failure. We have the frameworks. What we don’t have is the checklist.
The timing matters. The EU AI Act’s enforcement teeth for general-purpose AI arrive this August, high-risk obligations are phasing in behind them, and ISO/IEC 42001 is now showing up by name in third-party risk questionnaires. NIST’s AI Risk Management Framework has become the default answer for “show me you have an AI risk program” in North America. Add the OECD Principles, HITRUST’s AI assurance work, sector regulators like the FDA, and a growing patchwork of US state laws, and most enterprises are now operating under two or more frameworks simultaneously.
Here’s the part that surprises people: the frameworks themselves largely agree. Published crosswalks show substantial overlap between ISO 42001, NIST AI RMF, and the EU AI Act. An organization that builds its program thoughtfully can satisfy all three with a single set of processes and documentation. The problem isn’t the frameworks. The problem is what happens downstream, when those frameworks get translated into the questionnaires, audits, and attestations that land on real desks.
The Questionnaire Problem
If you’ve been on the receiving end of an AI security questionnaire lately, you know the artifact I’m describing. Hundreds of questions. Free-text answers. Prompts like “Describe how your AI system ensures fairness” or “Explain your approach to responsible AI.” These questions have three fatal flaws.
First, they can’t be answered with evidence, only with prose. And prose isn’t compliance; it’s creative writing. A vendor with a mature program and a vendor with a good technical writer produce indistinguishable answers. The exercise rewards confident fiction and punishes honest uncertainty. When I see a question that any vendor can answer without producing a single artifact, I know that question isn’t reducing anyone’s risk.
Second, they ignore the nature of the systems they’re assessing. As I wrote in the last column, LLMs are stochastic systems that are extremely difficult to replay and troubleshoot. A point-in-time attestation about model behavior is stale the moment a model version changes, a system prompt is updated, or a temperature setting moves. Asking “does your model produce biased outputs?” as a yes/no compliance question fundamentally misunderstands what these systems are. The right question is whether you can measure it, log it, and show me the trend.
Third, they don’t scale with risk. I’ve seen the same 300-question addendum sent to a vendor running a marketing chatbot and a vendor deploying clinical decision support. When everything is high-risk, nothing is. The EU AI Act got at least this much right: its four-tier risk classification exists precisely so that obligations scale with consequences. Most homegrown questionnaires have no tiers at all.
What Usable Actually Looks Like
I want to propose a different bar for AI compliance, and it borrows directly from the checklist lesson: a framework is only as good as its worst question. Before any question makes it into your AI assessment, whether that’s a vendor questionnaire, an internal review gate, or an audit program, it should pass five tests:
1. Answerable with an artifact. Every question should map to evidence: a log, a config, an eval report, a data flow diagram, an architecture document. If the only possible answer is an essay, cut it or rewrite it. “Describe your approach to model security” becomes “Provide your logged inference parameters (model version, temperature, top P, token limits) for your production deployment.”
2. Scoped to risk tier. Classify the system first, then ask the questions that tier deserves. A limited-risk internal tool gets ten questions. A system touching patient data or financial decisions gets the full treatment. Don’t send Annex IV-depth documentation requests to a chatbot vendor.
3. Measurable or binary where possible. “Do you run evals? At what cadence? What was your last pass rate on your safety benchmark?” beats “describe your testing philosophy” every time. Reviewers can score it, trend it, and compare it across vendors.
4. Decision-relevant. For each question, ask: if the answer came back bad, would it change our decision? If removing the question wouldn’t change any outcome, remove the question. This single test eliminates half of most questionnaires I’ve seen.
5. Mapped once, reused everywhere. Build your control set once, then use the published crosswalks to answer NIST, ISO, and EU AI Act asks from the same evidence base. One process, multiple regulatory readings. If your teams are producing separate documentation for each framework, you’re paying a triple tax on the same work.
The Questions That Actually Matter
If I had to compress AI vendor assessment down to a single card, it would look something like this: Where is the model deployed, and who is the upstream provider? What data flows in, and what flows out, and where is that logged? Where are copies of that data stored, for how long, and who can access them? What inference parameters are logged per request, and can you replay an incident? What is your eval suite, what does it cover, and how often does it run against production? Where are the human oversight points, and what can the system do without one? What is your incident process when the model does something it shouldn’t? How do you swap or change models, what testing gates a switch, and do downstream customers get notified?
Notice these are largely the same questions I argued for in the red teaming column: before deploying, know where the model runs, what inputs it processes, what the outputs look like, and how business risk gets revisited over time. That’s not a coincidence. Good security questions and good compliance questions converge, because both are ultimately about whether you understand the system you’re operating.
Standardize the Model Card
There’s one industry fix that would eliminate half of these questions before they’re ever asked: a standardized model card. I raised this in the last column, noting that model cards tend to offer some insights but measurements aren’t standardized across the industry, and it matters even more for compliance than it does for red teaming. Today, every model provider publishes something different: different eval benchmarks, different safety disclosures, different levels of detail on training data, different definitions of the same terms. That inconsistency is exactly what forces every downstream customer to run their own bespoke questionnaire, and forces every vendor to answer the same questions a hundred different ways.
Imagine instead a model card with a fixed schema: model version and lineage, training data provenance categories, where and how long data is retained, a common core of eval benchmarks with published scores, documented safety mitigations, default inference parameters, and a change log tied to every model swap or version update. SOC 2 didn’t succeed because it was clever. It succeeded because everyone agreed on what the report looks like, so one artifact could answer a thousand customers. The model card should be AI’s equivalent: produce it once, keep it current, and let it stand in for the first fifty questions of every assessment. Standards bodies are circling this idea, and both ISO 42001’s documentation requirements and the EU AI Act’s transparency obligations gesture at it. But until the schema is common, buyers can force the issue by asking for the same fields, in the same format, every time. Procurement pressure standardized SOC 2. It can standardize the model card too.
The key insight I want to share is this: compliance is an evidence problem, and for AI, evidence is an observability problem. The organizations that will sail through the enforcement wave now arriving aren’t the ones with the thickest binders. They’re the ones whose logging, evals, and documentation were designed so that any reasonable question can be answered in minutes with an artifact rather than in weeks with an essay. This is what I mean by timeless compliance. Frameworks will keep multiplying, regulators will keep diverging, and the models themselves will be unrecognizable in three years. But the principles underneath don’t move: know your system, log what matters, measure continuously, scale scrutiny to risk, and never ask a question you can’t act on. Those held true for surgical checklists and pre-flight cards, they held true for SOC 2 and ISO 27001, and they will hold true for whatever comes after the current generation of AI standards. Comprehensive coverage is a moving target. Simplicity, done honestly, is permanent.
This column is Part 3 of multi-part series on securing generative AI:
Part 1: Back to the Future, Securing Generative AI
Part 2: Trolley problem, Safety Versus Security of Generative AI
Part 3: Build vs Buy, Red Teaming AI
Part 4: Timeless Compliance (This Column)
Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay
