Book a call
Menu

AI Production Readiness Scorecard

Twenty questions on what has to be true before an AI pilot runs in production. You see your score on this page before you give us anything.

  • Yes: in place, written down, and someone other than its author can show it.
  • Partly: in place informally, or for some of the work.
  • Not yet: not in place, or you are not sure.

The score is calculated in your browser. With JavaScript turned off you can still read the 20 questions and their articles.

The 20 questions

Step 1 of 5 Group 1. Integrated

  1. The pilot reads and writes in the live systems it will use in production, not only in an export or a test copy.

    Why it matters: A pilot that ran on an export meets its first real connection in production. Integrating an AI pilot with the systems you already run

  2. For each connected system, we have written down what the AI may read and what it may write.

    Why it matters: Default access can be wider than the job needs. Same article.

  3. The AI acts under a named service role, so a change in a connected system shows who or what made it.

    Why it matters: Without a role, a record cannot say who changed it. Same article.

  4. We have decided, and tested, what the AI does when a connected system is slow, down or returns a bad record.

    Why it matters: The first outage should not be the first time anyone decides. Same article.

Step 2 of 5 Group 2. Secure

  1. The AI holds only the access its job needs, and a named person reviews that access.

    Why it matters: A wrong step reaches as far as the access allows. Agentic AI from pilot to production

  2. Our security team has reviewed the system, including the models it calls and the data it sends them, and open findings are closed or accepted in writing.

    Why it matters: A pilot that has not passed security review does not go live, and a written decision ends the wait. Why AI pilots stall

  3. We know which data goes to which model and where it is processed, and the business owner has approved it.

    Why it matters: Data location and model choice are decisions to make before go-live, not after. How to move an AI pilot to production

  4. Keys and credentials sit in a managed secret store with a named owner, not in code, prompts or shared documents.

    Why it matters: A credential in a prompt or a file travels with every copy of it. Production AI engineering

Step 3 of 5 Group 3. Governed

  1. A test set built from our own cases runs before every change to the model, the instructions or the content.

    Why it matters: Cases from your own operation show whether a change helped or broke something that worked. What are LLM evals and How to evaluate AI agents

  2. Actions that cannot be undone or that leave the company, such as paying, deleting, changing an access right or writing to a client, wait for a named approver.

    Why it matters: The control follows the consequence. Human in the loop vs human on the loop

  3. Every approval has a backup approver and a timeout rule: if nobody answers, the action does not run and the owner is alerted.

    Why it matters: A gate that waits for "someone" waits forever, or gets bypassed. Approval gates for AI actions

  4. For the decisions we chose, a time-ordered record shows what the system saw, what it did and who approved it, and a colleague can explain a decision from that record alone.

    Why it matters: A decision that is questioned has to be explained from the record. Define production for an AI system

Step 4 of 5 Group 4. Accountable

  1. One named business owner answers for the outcome of the system.

    Why it matters: When a result is questioned, one person explains it. Who owns an AI system after go-live

  2. One named technical owner runs the system, with a named backup.

    Why it matters: The builder should not be the only person who understands it. Same article.

  3. Both owners have signed a one-page definition of what "in production" means for this system.

    Why it matters: Without a written definition, production means something different to each person. Production AI engineering

  4. The people who use the system know who to call, and a stop path exists with a named person allowed to use it.

    Why it matters: A supervisor who sees a problem and cannot act on it is only a witness. Human in the loop vs human on the loop

Step 5 of 5 Group 5. Built to run

  1. We monitor response time, errors, availability and cost per request, with a named person on call.

    Why it matters: Monitoring tells you that something is wrong. LLM observability vs monitoring

  2. For each request we keep the input, the context given, the model and instructions in use, each tool call and the output, and the owners can read them.

    Why it matters: Observability helps you find why. Same article.

  3. A change to the model, the instructions or a tool is written down, retested, rolled out to a small share first and approved by the business owner, and the previous version is kept.

    Why it matters: A system that changes without a decision drifts. Production AI engineering

  4. A named person reviews a sample of real completed work on a schedule, and we know what happens when quality drops.

    Why it matters: A system that was good at launch can drift. Who owns an AI system after go-live

What does production ready mean for an AI system?

An AI system is production ready when it runs inside your operation, can be explained when it is questioned, and is owned by named people. Production AI engineering is the set of decisions and controls that make this true: how the system is evaluated, observed, connected to the systems you already run, controlled and owned.

How is the score calculated?

Each of the 20 questions is answered Yes, Partly or Not yet. Yes counts 5 points, Partly 2 and Not yet 0, so each group of four questions scores 0 to 20 and the total scores 0 to 100. Under 40 is Pilot stage, 40 to 74 is Part of the way to production, and 75 or more is Close to production when no group scores under 10. The calculation runs in your browser.

What does the report contain?

The report repeats your band and the 20 questions. For each question it gives the decision it needs and who usually takes it, with the article to read. It ends with a one-page production definition to complete and the order of work for your band.

What we store and why.

Your answers are scored in your browser. We record an anonymous event when the scorecard is started, completed and when a report is requested (score, band and the page you came from; no name, email, IP address or cookie), only to measure how the page is used. The result link contains your scores only, nothing that identifies you. If you request the report, we store your name, work email, company, role, scores and answers in our own systems to give you the report and, only if you ticked the box, to contact you about your result. Data is processed in the United States. We keep form details for up to 3 years after your request, and anonymous events for up to 12 months. To see, correct or delete your data, write to [email protected]. Privacy policy: www.ornsoft.com/privacy-policy.