The AI pilot is working well, what Governance do we need before we can deploy?

AI governance should not be an afterthought. Companies need to actively manage their AI, even if it is just embedded in commercially bought-in products, but for organisations looking to develop their own AI-based solutions, governance should be built in from the outset, not bolted on at the end. If the product has already been developed, adding governance between development and decision to launch can still
be done but comes with a greater risk of delay or rework.

Build governance into the product cycle

Traditionally, product development follows a design-build-test model. But in an AI environment, this needs to change into assess-design-build-test-evaluate [repeat].

Step 1: Assess

What are we trying to build? What are the potential regulatory, compliance,
reputational or other impacts of the development when deployed? For example, does an AI chatbot that estimates customer sentiment raise privacy concerns? What data will it use, and do you have the right to use it that way? What happens if an automation AI makes a decision – does the customer have an escalation path to a human to address any possible errors? Will the proposed solution impact jobs or job roles and how will that be managed? These sorts of questions ideally need to be
asked at the outset, with the correct senior staff in the room, and the answers documented. The later you ask the questions the higher the cost of any adjustment in direction and very late changes are generally very expensive.

Step 2: Design

This is familiar to normal software development but there are additional considerations. AI is not 100% reliable or repeatable. Design needs to be defensive from the outset, taking into account that the AI may do unexpected things. This means guardrails, monitors, triggers, trip-wires and sandboxes to ensure the development is, and remains, well contained. Ensure that the development team is experienced with this technology – it is very different to developing a consumer banking application for example.

Step 3: Build

Again, not dissimilar to conventional development but with the added complication that the AI is probably a cloud service integrated into your product. The system needs to be robust, for example if the cloud service stops working for a period. The development also needs to include measures to avoid common cybersecurity attacks– for instance a bot continuously calling a customer care helpline in order to run up the AI bill (a “denial of wallet” attack), or “prompt injection”, where hidden instructions trick the AI into doing something it shouldn’t.

Step 4: Test

Once more, similar to conventional software but not the same. In conventional software it is possible to define an exhaustive set of tests that will all run in exactly the same way every time. With an AI, the response from the model is based on probability, which means that the answer may vary, either subtly or substantially, from run to run. The result is that you cannot test quality into the product; it needs to come from design. Testing becomes statistical: run large sets of realistic questions, measure how often the answers are acceptable, and deliberately try to break it. And apply guardrails so the system is resilient to the AI occasionally providing different answers to the same question.

Step 5: Evaluate

Alongside the initial assessment, this is one of the most important phases and often where projects stall. The project works and delivers what was asked. But seeing it in action raises new questions that were not initially considered. How does automating this process affect the other processes around it? How about staff training? How will customers respond? Ideally all of this would have been covered in Step 1 but that is not always possible (it’s often hard to evaluate something if you have never
experienced it). Which means going around the loop again: reassess from where you are today, and put any changes through the same cycle.

A loop, not a waterfall

In software engineering, a waterfall development is one that has a clearly defined start, middle and end. It works well for a project with a 50-page specification document with few uncertainties. They are popular with management because they often come with a notionally fixed price and a clear return on investment. But like many fixed price contracts (building a home or a railway), a lack of initial investigation results in cost and time overruns that can exceed the original budget.
By definition, AI projects do not have certain outcomes.

This need not be a worry; often the uncertainty means the AI provides a much better solution than was initially envisioned. But it does mean you need to develop it like a painting. Put in place the sky, sea and land to begin with and then add the detail. Do not start on the left-hand
side and work across. You are a creator, not a laser printer.

Iteration is a strength, not a weakness

Iterative design allows you to address the questions at the earliest possible point in the project, when they are cheapest to fix. A linear project may look more predictable but it may not deliver as promised. Keep each loop short (a few weeks) and buildtowards the project you want rather than the one you can specify at the outset.

AI governance should not be an afterthought. Companies need to actively manage their AI, even if it is just embedded in commercially bought-in products,

How this relates to governance

Governance touches on every stage of the development cycle. The most important is the first – imagine the project is complete and consider the issues it brings in terms of reliability, resilience, regulation and change management. Then design the project with governance in mind – how will you be able to audit the decision-making? How will you fall back to a manual process if something goes wrong? Where is the human decision-making (human in the loop) and realistically can they review the 35 th loan application that day? Before launch, a named senior person should formally sign off that these questions have been answered. And governance doesn’t stop at launch: monitor how the AI performs in live use, and re-test whenever your supplier updates the underlying model – its behaviour can change overnight.

AI governance should be the first step, not the last – and it carries on after launch
Know what you don’t know and design accordingly, using iterative development rather than a rigid process
Ensure all relevant stakeholders are involved from the outset and record your decisions, in order to prevent other parts of the business (for example, Risk, Compliance, HR or Legal) from blocking your project just as you prepare to go to market.

To explore further

Take a look at our Resources page with many  other helpful articles on AI and governance

Try our tools and solutions

Discover Akili’s tools and solutions designed to help you understand,
assess and govern AI with confidence.