#11 When AI breaks: systems that learn faster

At our recent roundtable, someone remarked: “It’s not my job to fix the AI while I’m trying to use it.”

It raises some interesting questions. Whose job is it? How do they know it’s broken in the first place? Where and why did it break? How can it be fixed, and what do you do in the meantime?

These are all questions we’ve been wrangling with in a recent project with a government department as they look to use AI to handle an increasing backlog of digital evidence. This work has only reinforced our opinion that the responsible use of AI, especially generative AI, requires a systems perspective. Because AI has a ripple effect: it changes how the system behaves.

Understanding if the AI is working as expected is not just about getting some evals in place (although this is still important!). It involves subject matter experts constantly checking the system’s outputs for accuracy and appropriateness. 

It involves asking: Are humans in the system using the features as intended, or are there any signs of automation complacency, accountability confusion, or skills fade? It requires constant scanning for misuse – malicious or unintentional.

All of these processes require organisations to set up multiple layers of monitoring across the stack. That stack includes the model, the AI system, the digital tools that use the AI system, and the processes and people using the digital tools. Without this holistic approach, it’s simply impossible to understand if the system is working as expected.

Of course, none of this matters unless you can answer the question: What do you do when something unexpected happens? How fast do you need to act? Who owns the response? What’s the rollback plan?

AI will go wrong. And it will go wrong in new and unexpected ways. Knowing this and planning for it will allow you to respond better and faster. It will also dramatically increase the chances of your new AI products and services moving from small experiments to strategic successes.

For 10 years, Projects by IF has been helping large organisations build customer-facing services that scale safely, earn trust from the start and deliver long-term impact.

We prototype, test, and launch AI products and services that people believe in and want to adopt, while helping organisations change the way they work in the AI age. 

Learn more or talk to us.

What we’ve been up to in the studio

From prototypes to pilots: what does it take to scale AI-enabled services? 

We’re wrapping up prototyping activities and supporting cities on the Bloomberg Philanthropies City Data Alliance (CDA) AI Track to plan their next steps at a product level, while looking at which strategic enables are required to support AI at scale. And we’re starting to work with even more cities across North and South America to embark on their AI-prototyping journey.

Designing for trust in the age of AI

Valeria flew to Spain to join leaders from design, marketing and behavioural economics at a multinational bank to share our learnings and insights on designing for trust in the financial sector. We shared why trust matters and how you can demonstrate trustworthiness by translating brand intentions on trust into the customer experience. 

What we’ve been reading

The Government announced BritCard, a mandatory digital ID for Right to Work checks. Sarah Gold responded with a set of guardrails and trust signals needed from day one if BritCard is to have any chance of launching successfully - convincing the 2.5 million people who have already signed a petition against it. Read the article.

A new paper explores large language models as evaluators, not just generators. It lays out structured methods to assess reasoning, summarisation, and judgement, offering a blueprint for using AI to check AI. Read the paper.

The Open Knowledge Foundation has published a new field guide, The Tech We Want. It’s a practical resource to consult before you start building – helping teams think critically about governance, ethics, and social impact from day one. Read the guide.

Albania has appointed Diella, the world’s first AI-generated government minister, to oversee public procurement. The intention is to eliminate corruption - the reality involves a lack of human oversight and the potential for manipulation. This move raises important questions about the role of AI in governance and the balance between innovation and accountability. Read the article

Coming up

This November, we’re hosting a panel event in London: “How to compete in the AI age without breaking trust”. We’ll cover how organisations can avoid the “fail fast” trap and turn adoption into real outcomes. Tickets are going fast: register here.

Until next time, 

— Mark and the IF Team

Find us on LinkedIn and Medium for more insights and updates.