Secure and fair AI systems for citizens: Program Results

21st Oct 2021

The expansion of artificial intelligence (AI) has opened the doors for advancements and improvements in almost every domain of human life. However, the development of AI comes at a cost – the methods are not bulletproof, and may be thwarted by a number of issues, among them two major ones. One regards the technical side of it and is related to the fact how many new cyberattacks keep emerging, and consequently, require employing adequate countermeasures. The other set of challenges to the successful proliferation of AI are of the humanistic nature, and relate to the concepts such as trust, fairness and other social challenges. Among them, there is also the question of explainability, i.e. explaining the outcomes of an algorithm in a transparent way, so that human operators can understand where the decision is coming from. The need to answer all of those issues sits at the heart of the SAFAIR (Secure And FAIR AI systems for citizens) program. Its primary and most ambitious goal has been the provision of trustworthy AI solutions; the trust being based on their being secure, reliable and built with explainability. In other words, the goal was to make the AI-based solutions more robust against being tampered with by threat actors, and at the same time, to ensure the decisions made by AI remain fair and explainable. This in turn would result in creating resilient systems that citizens can rely on. 

The work was divided into five steps. Firstly, a set of systematic threat analysis tools was developed, aimed at providing support for AI-based architectures. On top of that, the SAFAIR threat knowledge Base was built, which has been made publicly available.  This gives both the users and the developers the chance to understand the threat landscape better, to be aware of the vulnerabilities, and the possible consequences of various cyberattacks and other threats to data integrity. By such an approach, threats are tackled starting from the early design stages. 

The next step was coming up with reactive security tools, based on the identified threats, in order to make the AI solutions more resilient against them, having security and privacy in mind. A number of various countermeasures have been employed to ensure the security of the systems, which at the same time do not affect the system’s functionalities. 

Some of the defensive solutions that SAFAIR has proposed and implemented are:

  • a novel approach handling adversarial attacks by using Prediction Similarity
  • a novel way to detect adversarial attacks using a Neural Activation-based Adversarial Attack detector
  • Model’s Behaviour Defence
  • Neuron’s Behaviour Defence, and more.               

For example, one of the above-mentioned works concerned securing the machine learning methods applied in cybersecurity against adversarial attacks. In order to do so, an algorithm was designed, able to detect evasion attacks on intrusion detection systems. This method was then tested on the strongest adversarial attacks known at the time: Fast Gradient Sign, Basic Iterative Method, Carlini and Wagner attack, and Projected Gradient Descent. The reported results show promise; they suggested that it is possible to create an effective adversarial attack detector that does not influence the classification results of the model it protects. 

All the methods and tools that were introduced were then applied in a variety of domains; for example, VICOM worked with medical images, TCS scrutinized malware in PDF files, and ITTI worked on defending AI algorithms included in Network Intrusion Detection Systems, and on the preprocessing pipelines to robustify Face Reidentification against adversarial evasion attacks. 

In addition to this, a competition was held, the goal of which was to come up with an array of defense mechanisms and validate SAFAIR approaches. 

In parallel, work was performed to enhance the explainability of AI. This is especially significant with critical tasks becoming automated; the ability to interpret and understand the reasons for decisions boosts the confidence of the people using AI-based systems. 

A number of mechanisms enhancing explainability and fairness were designed, amongst which the Hybrid Oracle-Explainer xAI system, a state-of-the-art solution able to deliver high-quality predictions and use surrogate-type explanations to and present them in a comprehensible way. Methods to apply explainability on the BERT-based AI models used in Fake News detection were developed, and local explanations of machine learning models with Shapley Values were explored.

Considerable efforts went into putting mechanisms in place so as to reduce bias in the decisions made by the AI-based systems, thus making them as fair, and discrimination-free as possible. 

Below there are some papers which explore the subject further: