In order to use AI systems with confidence and trust, they need to be not only reliable, but also secure. This aspect of the development and deployment of AI systems is highlighted by the requirement for their technical robustness. This requires that AI systems are developed to behave reliably and in the way we originally intended. Put simply, that AI systems do what they are supposed to do while not doing what we do not want them to do, particularly if this could have a negative impact on our safety, health or fundamental rights. It thus includes criteria that largely overlap with cybersecurity requirements, such as resilience to attacks or having back-up plans in case of a major event, but it also includes issues of accuracy, reliability or reproducibility.
Like any other software system, AI systems must be protected and resilient against misuse and potential attacks. These may be aimed at obtaining or degrading sensitive data, but also the model itself, or they may involve attacks on the entire infrastructure. However, unwanted changes in the behaviour of an AI system can also occur unintentionally, for example if such a system is exposed to unexpected situations that it is not prepared to face (an example of a deployment could be the Tay chatbot), whether this is in the case of unintended use or outright deliberate misuse. It is therefore important to preemptively analyse such risks and at the same time set appropriate measures to prevent potential harm at every stage of the development or deployment of AI systems. Of course, the level of precautions also depends, however, on the potential extent of the risk that the AI system poses to humans and the specific context in which such a system will be deployed.
Another option highlighted by the requirement for reliability and security is the implementation of a back-up plan. This may mean that AI systems switch to a safer or more robust mode if necessary, or that they require intervention by a human operator before continuing to operate, or even shut down completely. All of this can also happen to preserve the reputation of the company or its customers.
An equally important aspect of trustworthy AI systems is their accuracy, or the negative consequences that can result from their inaccurate outputs. This refers to the ability of the systems to form valid predictions, recommendations, or decisions. We should therefore be transparent in measuring and comparing their accuracy and communicate openly any inaccuracies or weaknesses identified, but with due regard for the aforementioned requirements for security and resilience to abuse.
Precise and clearly defined accuracy criteria are also closely related to the issue of reproducibility. The outputs from AI systems should be,reliable, taking into account in what context and with what input data it works, or with respect to specific testing and deployment conditions. At the same time, researchers should be able to reproduce the operation and behaviour of such systems, for example for auditing or functional verification purposes.
