1. Key Policy Questions and related issues:
Even black boxed systems can be explained, given the choice of the right type of explanation. However, the most important aspect to be explained is not technical, but rather political: what is the purpose of the system and what are its success factors ("this is often the most striking blind-spot" – Hong Qu).
It is crucial to give affected individuals access to justice -- a chance to contest unfair or erroneous decision – rather than mere explanation of why the system produced certain result.
We must take a human-centred design approach to monitor for disparate impact which harm real people, and second order externalities that might affect vulnerable populations.
2. Summary of Issues Discussed:
Practical approach to explainability: how can it be achieved?
While there are test and edge cases for hardware and software, there is no "testing standard" for AI. Speakers agreed that responsibility of the impact of AI systems needs to start inside of organisations that produce them. AI system developers should perform tests to detect potential flaws and risks ("surface unintended consequences" – Hong Qu), before implementation and they should make results of these tests public (e.g. using the model cards framework).
Auditing AI systems is only a starting point to more substantial reform. In many cases, legal regulations that we need to challenge the corporate and government uses of AI are already in place, they just need to be enforced or pursued through litigation. Authorities and courts are prepared to handle confidential information and therefore can be employed to audit/investigate AI systems. They also have the democratic mandate to do so.
Value of explainability for people affected by AI systems
Explainability as we see it today is not always actionable for end users. Most stakeholders don't need to understand this level of technical information. Also, the way field experts talk about explainability often misses the bigger picture. It tends to focus on the interpretation of individual result (i.e. how the AI system went from general features to individual decision), but ignore other essential features of the system that are equally important, such as its robustness and safety or fairness.
Different kinds of explanations of how AI decisions are made might not have that much effect on whether people think they are fair; their intuitions about whether it's ok to make a decision about someone using a statistical generalisation tend to override any differences between different ways of explaining those generalisations.
