Opinion
X by design
From privacy to explainability, trust and robustness
In the design process of any system, privacy-by-design means that privacy is something that you take into account from the beginning and not as an afterthought. It implies that the performance of the system with respect to privacy is taken to be a component of the overall performance criteria. How little of the users’ data can you expose to outsiders, to the service provider and to other stakeholders, without compromising the desired service? Within privacy, this approach is a no-brainer. After a little contemplation, it is plainly obvious that the design of a system that has good privacy requires privacy-by-design, to such an extent that it is required by the European privacy laws, known as the GDPR.
Recently, I realized that we should apply similar approaches also to things like explainability, trust and robustness, or explainability-by-design, trust-by-design and robustness-by-design.
Explainability by design
First, explainability refers to the transparency of causal relationships in a model. That is, suppose an AI decides to turn on the heating in a home, then the system is explainable if the user can find out what the trigger for turning on the heating is. For example, the system could decide that if the temperature is below a threshold, then the heating is turned on.
In such a toy example, explainability is thus trivial. There exist a rule and it should be transparent to the user. Real-world AI is however much more complex. There are many variables and the relationships between input variables and the output can be highly non-linear. In fact, the whole purpose of AI systems is to find causal relationships from data that are so complex that simple rules are not available. We can thus not demand that the rules should be transparent. Instead, for each of the input variables, we can label their respective weight or importance in choosing a particular outcome. For example, “because of your good grades during the last year at the university, I decided to hire you”. That is, even if there are a large number of criteria for choosing whom to hire, “grades” was labelled as the trigger. It is thus a post-hoc approach, where we make a reverse inference, to determine which features of the input were important/decisive for the output.
Explainability by design would then refer to the design process, where we think about how to enable explainability from the start. For example, if we design an end-to-end machine learning system that takes raw data as input, then it might be difficult to assign meaningful labels of importance. Say, if we have a voice user interface that takes the microphone signal as input, it does not make sense to assign importance to particular samples of the microphone signal. They are meaningless to the user. Instead, we would need some other feature representations, either parallel data sources or intermediate levels of the system, that would provide heuristically understandable explanations of causal relationships. For example, “your voice is hoarse, so I assume you have the flu”. An explainable design thus requires that you can extract qualifiers, which describe heuristically meaningful features, to assign causal relationships.
The design process that takes explainability into account, thus necessarily includes the study of qualifiers, about how they are chosen or generated. If we have a source for such qualifiers, I would expect that it is feasible to extract their causal importance. For example, we can use back-propagation to study the sensitivity of qualifiers with respect to the input. Often, such qualifiers are however not available and their generation would require structural changes to the AI system. In practice, we would thus have to retrain machine learning modules, which is a labour- and computation-intensive process. It is, therefore, better to use an explainability-by-design approach such that the qualifiers and reverse-inference methods are included in the design from the beginning.
Trust by design
Here I would define trust as the feeling that one can predict the outcome of events outside direct control. Thus we should split trust into two parts, to the trustworthiness and to the feeling of trust, where the former refers to consistent behaviour (predictability) and the latter to the feeling (belief) that we can predict, irrespective of the actual level of predictability. Both are obviously important. When flying an aeroplane, we both want to feel safe (feeling of trust), but also that we are actually safe (trustworthy). The same applies to AI systems.
The requirement of consistent behaviour is a task with which AI designers are likely already familiar. It refers to making sure that the whole range of expected inputs gives sufficiently accurate outputs. It refers to the training, testing and verification methodology used, such that gain evidence of consistent performance.
Perhaps the more interesting part is thus the level of trust which users feel when using the AI system. Clearly, this is a task of user interface and experience (UIX) design. Again, UIX designers need to be involved in the design process (sic) from the beginning. For example, in the choice of qualifiers for explainability, we need to consult UIX designers. How would users feel if hiring choices are based on the amount of hoarseness in one’s voice? Clearly, such choices are (can be) also ethical and moral choices. Surely trust by design is therefore well-motivated and important.
Robustness by design
A classical approach for evaluating AI systems is to test their performance on unseen (never-before-seen) data, which was not used for the training of the system. It is a robustness criterion, in the sense that sufficiently similar datasets should give similar outcomes (both inter- and extrapolation). The question is however whether robustness can be designed beyond that?
A more fundamental question is to define what robustness means. A heuristic definition would be that the outcome of the inference should not be overly sensitive to small changes to any of the inputs. In mathematical terms, it would be beneficial if several independent dimensions of the data would point to the same outcome. That is, in a hiring decision, it is not good if the decision is based solely on the grades of university courses, but also other sources, like the employment history, should support the same outcome. That means that small changes in grades would not flip the decision. An obvious challenge is that there will usually always be borderline cases where small changes will flip decisions.
With testing, we can thus estimate the robustness of a system and with dropout-methods we can train a system such that it is less dependent on particular dimensions. I’ve also recently seen some work on enforcing continuity and smoothness criteria on outcomes, which clearly improve robustness. Still, I feel like I’m missing something here. I feel as if there should be some other, better way of making sure that decisions are robust, such as based on a majority vote of independent indicators. Here I see the largest practical problem in defining independence since classical definitions are based on linear operations, but AI systems are non-linear. In any case, my non-scientific gut feeling is that these are questions that must be again included in the design process from the beginning.
Summary and Conclusions
I feel that I’ve arrived at a circular argument. Clearly, if we want to have robust and explainable systems which we can trust, then we have to design them to be such from the beginning. By extension, we could actually apply an ethos of ethical-by-design, which would encompass robustness, explainability, trust, privacy etc. In plain language, we could then just say that we design good systems. How is that then a different objective from what we already do?
For one, I think that framing the design of AI systems as holistic design problems would be a significant improvement. It is not only about how accurate a hiring-AI is with respect to a training dataset. It is not only about how many customers click on the banner chosen by an AI system. AI systems are not isolated systems, but they interact with the greater society. Within hiring-AI’s, we need to evaluate also the society-level responses to the system and with ad-recommendation systems, we need to evaluate their effect on vulnerable populations, on democracy, on societal and economic justice and diversity, and so on. A mighty task.
The responsibility of good design lies on the designer (sic). While the above-described task is hard, the responsibility lies on us, the designers. Always when we design, but especially when publishing results, we should frame the goodness of the system in terms of its holistic performance. Similarly, when reading about AI, we should make questions and demand answers about holistic performance. I’m not sure if I can even think of an option, the consequences could be grim, we have to do good design.
