AI production readiness checklist: define production for an AI system on one page

An AI production readiness checklist only works if everyone agrees, in advance, what production means. So start with one page that fixes six things before the build: the scope, the success criteria, the evaluation set, the security setup, the approvals and the go-live plan. The accountable business and technical owners sign it. From then on, “is it ready?” has an answer that everyone agreed in advance.

A production definition is a one-page document, signed by the accountable business and technical owners, that states what production means for an AI system: what it does, how it is measured, how it is secured, what a person approves and when it goes live.

This article gives the template we use at OrNsoft in every forward-deployed engineering engagement, and the checklist we run on each section before anyone signs.

Key takeaways

• Production is defined on one page, before the build, not after the first error.
• The page has six sections: scope, success criteria, evaluation set, security, approvals, go-live.

Why does an AI system need a production definition?

Because without one, nobody agrees what “good enough” means, and every error reopens the debate.

A pilot is judged by impressions: it worked on the examples people tried. Production is judged by the operation: it has to work every day, on live data, under the rules that apply.

The gap between the two is where pilots stall. Integration, security review, measurement, approvals and ownership each need a decision. The production definition records those decisions on one page.

It also protects the project. When an error appears in week six, the question is no longer “can we trust this?” but “does this meet the criteria we signed?”. That is a measurement, not an argument.

The idea is not new in software operations. Site reliability engineering uses a production readiness review to identify what a service needs to run reliably. For machine learning, Google researchers published the ML Test Score, a rubric of tests and monitoring needs for production readiness. The production definition applies the same discipline to an AI system, and adds the business owner’s signature.

What does the one-page template contain?

Six numbered sections, a header that names the owners, and three signatures.

Here is the same template as a table. The fields in brackets are filled in with your owners.

Field	What to write
System	[System name] · Organization: [Organization] · Business owner: [Name, role] · Technical owner: [Name, role]
1. Scope	[What the system does, for whom] · Out of scope: [What it never does]
2. Success criteria	[Measurable criteria the system must meet, and the conditions under which they are measured]
3. Evaluation set	[Which examples, labeled by whom], frozen at signature
4. Security	[Where data stays] · [How people sign in] · [What is logged]
5. Approvals	[Which decisions a person makes] · [What the system may never do on its own]
6. Go-live	[Date] · [How long the system runs alongside the current process before cut-over]
Signatures	Business owner · Technical owner · Engagement lead

One page is a constraint on purpose. If a section needs three pages, the decision behind it has not been made yet.

The AI production readiness checklist: what to verify before anyone signs

Run each section through these questions. A “no” means the page is not ready.

1. Scope

• Does the scope describe the task in the words of the people who use the output?
• Is there an explicit “out of scope” line, especially for decisions with financial, legal or safety impact?
• Is it clear which systems the AI reads from and which it writes to?

2. Success criteria

• Is every criterion measurable, with a number or a yes/no test?
• Do the criteria cover quality, speed and the cases the system must never get wrong?
• Has the business owner agreed that meeting these criteria means the system is useful?

3. Evaluation set

• Is the evaluation set made of real historical cases, not invented examples?
• Is it labeled by people with the authority to say what the right answer is?
• Is it frozen at signature, so results stay comparable before and after go-live?

The NIST AI Risk Management Framework, a voluntary framework, makes the same point in its core: test sets, metrics and the tools used for evaluation are documented (MEASURE 2.1).

4. Security

• Is it written where the data stays and who can reach it?
• Does sign-in use the organization’s own identity, with role-based access?
• Is every action logged, and does the log say who or what acted?

How OrNsoft approaches security and data is on our trust page.

5. Approvals

• Is every action with real consequences listed, with who approves it?
• Are the approval gates part of the system, not a manual habit?
• Is it explicit what the system may do autonomously and what requires approval?

The NIST core asks for the same: processes for human oversight are defined, assessed and documented (MAP 3.5).

6. Go-live

• Is there a date?
• Will the system run alongside the current process, and for how long?
• Is it named who runs the system after handover, with a runbook?

7. Signatures

• Have both owners read the page in full?
• Is the engagement lead named, and accountable until the system runs?

Signatures turn the page into a decision with owners. NIST’s GOVERN 2.1 asks that roles, responsibilities and lines of communication for managing AI risks are documented and clear.

Who should write the production definition?

The engagement lead writes it, with the owners, during the diagnosis.

• The engagement lead knows what can be built and measured.
• The owners know what the operation needs and what an error costs.

It should never be written by one side alone:

Written by	What goes wrong
Engineers only	It measures what is easy to measure.
The business only	It describes the outcome without the conditions that make it testable.
Both, with the engagement lead	The criteria are useful to the operation and testable by the build.

The engagement lead is the named senior lead, responsible until the system runs. For the role itself, see what a forward-deployed engineer does.

What happens when the page cannot be written?

Sometimes the diagnosis shows one of three things:

1. The data is not there.
2. The owners disagree on the goal.
3. The process the AI supports is about to change.

Then the honest answer is that the pilot is not ready.

At OrNsoft we treat that as a valid outcome of the diagnosis. If we cannot write this page with you, we tell you the pilot is not ready for production, in writing, with the reasons. It is the cheapest moment in a project to learn it.

How does the production definition fit into the rest of the engagement?

The page is the first gate of four.

In a forward-deployed engagement at OrNsoft:

1. Diagnose (first two weeks). We read the pilot, meet the owners and write the production definition. Gate: the owners sign it.
2. Build and integrate. In your codebase and target deployment environment. Gate: security review passed.
3. Prove it in production. The system is measured against the criteria. Gate: criteria met, in writing.
4. Go live and hand over. Cut-over, monitoring, runbook and training. Gate: your team runs it.

Weekly written progress against these gates keeps the owners informed without extra meetings. The full sequence is described in how a forward-deployed engagement works, week by week.

OrNsoft is an advanced AI engineering firm, founded in Florida in 2009, with applied AI in its software since 2015. We build our own AI products and bring client AI systems from concept or pilot to production, with a named senior lead responsible until it runs. More on that in seventeen years in production.

What happens next?

If you have a pilot and want to know whether it is ready for production, see how forward-deployed engineering works, or book a call. One call is enough to know whether we can help.