The importance of Explainable AI in Software 2.0

The short answer is this: in small portions in every lifecycle step of software 2.0 production and consumption alike.

Data curators will need to use tools to help them in their labour. The labelling process will involve many iterations, conflicts and not clear answers. Labelling and conflicts will sometimes be hard to resolve. To make their job easier, the tools will assist them with analytics on the datasets, insights and suggestions. If they are to trust AI to assist them, they too, need to understand where the suggestions come from.

Machine learning teams will have to include interpretability in their best practices. ML researchers will no longer be able to base experiment results on intuition. They will need to communicate unexpected behaviours and tackle them by either finding specialized data sources or creating synthetic data. They can only know what data to look for, if they know how the model handled the specific use case.

Quality assurance teams need to test the feature reaching the customers. To do so effectively they will need to understand what is happening under the hood. What are the patterns the model “sees” and produces the specific outcome. Bizarre model outcomes will need to be explained to be fixed. Tuning to perfection will require looking deep into the “thoughts” of the agent. Google has even published an exhaustive checklist for ML quality assurance in the form of a paper.

In business, nothing will be taken for granted. Transparency is key in gaining the trust of customers and if we are to automate regulated decisions, reasoning will need to come along with the conclusion. XAI is going to help customers and companies by educating them on how to reach the desired result in applications. XAI could even create new cross-selling or upselling channels for businesses.

Customers and consumers have already made AI part of their lives. As it will penetrate our every-day more and make things easier for us, we will be asking more and more questions. In order to let AI take responsibility of serious tasks, human-machine interactions will need to be trustworthy. As Bjarne Stroustrup (creator of C++) puts it “[software 2.0] … it is not good enough for life-threatening situations”. Doctors will not just trust machines giving predictions with no reason behind them, even if they tend to be correct. Sometimes even 99% accuracy won’t cut it.

Ethics is bound to play a major role in the development of AI. Companies need to be careful on how they use automated systems. Apple, with an Apple Card incident, and AWS, with a recruiting incident, have already been victims of AI discriminating against women because of low quality and unchecked datasets. Black box systems no longer qualify for consumer use.

Engineering teams are already beginning to create tools to help data curators. These tools mainly focus on data labelling, versioning and storing. Quality of datasets will be of utmost importance. Machine learning teams will need to move fast, experiment and communicate their results to business units. New initiatives in organizations or product features will be dependent on the outcome of these experiments.

Business decisions are taken based on data. Management will now have to pick whether to do data annotation in-house or outsource it. Tesla is already using such methods to produce data that their autopilot learns from. They already have teams of people whose job is to instruct machine learning models what road lines look like in the real world. Their data are not yet mature enough for their autopilot to be fully autonomous, so their business decisions are essentially taken based on the current status of their data quality, quantity and model behaviour. Data, technology and talent will be the driving force of strategic partnerships and acquisitions. ML is becoming an expensive investment.