Explainable AI – or XAI – is a relatively new research area that hopes to ‘open the black box’ on deep learning neural networks, complex algorithms and probabilistic graphical models.
Artificial intelligence systems that can explain their decision making process in human terms are now the subject of intense research by software and cloud vendor Oracle, the company’s senior vice-president of data-driven applications revealed to Computerworld yesterday.
“One thing we don’t make a big call out to is that we have a dedicated research team at Oracle called Oracle labs, mostly PhD computer scientists. And we have a lot of research going on that we don’t tend to advertise very much in those research groups looking into that specific research area,” said Clive Swan on the fringes of Oracle’s Modern Business Experience event in Sydney.
“It remains a big area of academic research. That problem is…very difficult academically to solve in some cases, and frankly varies from algorithm to algorithm.”
The aim of XAI research – which is being carried out by the likes of the Defense Advanced Research Projects Agency (DARPA), an agency of the US Department of Defense – is to give machine-learning systems the ability to explain their rationale, characterise their strengths and weaknesses, and convey an understanding of how they will behave in the future in a way that is understandable and useful to end users.
Many believe such systems will give industry, governments and the public greater confidence in using them.
“Explanatory AI is actually something we’re actually looking at and trying to work on,” Jack Berkowitz, Oracle vice-president of products, data science and adaptive intelligence, told Computerworld.
“We have some ability to explain that rationale. Sometimes people definitely need that… I think it’s important, I think it’s difficult. And I haven’t seen a great system yet that does it.”
Part of the problem with AI systems lies in the understanding of users, Berkowitz added.
“Some of the products do need to give that causation right…you need to explain but you also need to explain it in a way, because you don’t know if the person consuming actually understands what that is either. Then you need to think – do I have an informed and educated person on the other side? It is a two way partnership that has to be there,” he said.
At the summit, the company said it had upped its annual research and development budget to $5.4 billion.
Oracle rival Amazon Web Services hinted they too are exploring XAI. Speaking to Computerworld last month, chief architect Glenn Gore said there would eventually be some ‘explainable’ element to AWS’ AI products.
“Right now no. You just put data in and get attributes out,” he said. “As it evolves over time, being able to understand that decision making process to a certain level will be there.”