Roadmap Timeline

Secure and Fair AI Systems for Citizen

2022-2026

Problem description

The proliferation of Artificial Intelligence systems in contemporary lifestyle brings about both astonishing benefits and brand-new challenges for society. While the gains and the prosperity delivered by AI are abundant in all walks of life, starting from most obvious ones, like image recognition, search engines, recommender systems, autonomous systems, including vehicles, to less obvious uses, like cybersecurity. The widespread adoption of AI does not consider that those algorithms were developed not taking into account the adversarial nature of real-life implementations. Thus, an array of problems emerges. First and foremost, the bulk of above-mentioned algorithms have a black box nature. This means that even though the insights provided those methods are meaningful and valuable, no one can easily explain how exactly the AI came to its conclusions. Every machine learning model, prior to applying it, has to be trained. The training can be run in any of the following three ways: supervised, unsupervised and semi-supervised. Each of them has its advantages and drawbacks and is used in different applications. While the ML algorithms invariantly fit the presented data, it is a challenging task to try to explain how specific data affects certain aspects of the algorithms, which then translates to the end result. One of the facets of the SAFAIR program attempts to address the situation by enhancing the explainability of AI. Secondly, methods exist that allow to compromise AI itself in several ways. A knowledgeable individual can influence the way an AI classifier judges a specific data point, thus evading detection. A malicious user could also provide a series of inputs in the training, or re-training phase of a classifier – in other words poison the data – to make the algorithm behave in a way that is beneficial to the adversary. Thirdly, a trained AI setup constitutes a major expenditure of expert time and therefore company resources. This makes an AI model a valuable intellectual property. There are ways, however, to fit one classifier to the output of another classifier, essentially stealing the original algorithm. Last, but not least, any bias on the AI part, especially in socially sensitive areas, could relatively easily seed distrust to AI technology among the general public. In the midst of all that, there are new cybersecurity challenges that gain ground recently. With the universal danger of cybersecurity breaches, enhancing the cybersecurity condition and detection algorithms is of absolute importance. Malware is now identified as the stern menace for commercial and critical IT systems, as well as for the general public. Malware, however, is adequately comprehended and can be dealt with sensibly well. A more menacing challenge arises, stegomalware and the use of the information hiding techniques by cyber-criminals.

Research aspect: Contemporary threats to AI systems need to be investigated, and suitable countermeasures need to be developed. An in-depth analysis of current adversarial threats needs to be performed. As the threats evolve, the ability to address the needs to keep up. With no adequate measures for AI explainability, AI fairness and most importantly AI security, all of those aspects require suitable analysis. Defensive and preventive mechanisms need to be established. Along with improving the robustness of AI itself, research on new cybersecurity threats, like information hiding and ransomware is in demand.

Industrial demand: Every industry relying on AI technology is now vulnerable to adversarial attacks; this includes critical, sensitive domains, like automotive, government, medical fields, security-related, etc. Providing secure and explainable AI systems would increase trust in these kinds of systems, allowing further adoption, and preventing possible adversarial intrusions, hijacking of algorithms, or breakdowns. Risks are related to the various classes of assets. Structures like payment systems in the financial arena, embedded systems, cloud computing services and systems processing personal data are especially exposed to the danger of cyberattacks.

Social aspect: The wide audience needs to trust AI solutions to rely on the decisions inferred from data. The possibility of manipulation of AI breaks this trust and makes the whole big data ecosystem unreliable. Thus, AI resilient to adversaries is necessary. Appropriate use and re-use of data are mandatory for AI systems to continue to flourish. Thus, setting up systems to make AI compliant with current and upcoming data-related legislation is of utmost importance. Furthermore, establishing a track record of what is perceived by the general public as fairness with regards to how AI operates has the potential of accumulating trust to those kinds of solutions.

Final goal:
  • Enhanced explainability and better threat understanding in AI context
  • Systems using AI more reliable and resilient
  • More effective methods and tools for analysis of security threats for AI systems
  • A set of techniques and solutions for AI systems protection
  • Systems in place to ensure fairness of AI systems
  • Defensive and reactive mechanisms geared towards novel cybersecurity threats
  • Cybersecurity systems being able to detect stegomalware


Benefits for EU: This kind of technology could provide EU AI industry a leading position on the global market, given the unique selling proposition of the only secure AI on the market

Domain (JRC Taxonomy): Theoretical Foundation, Human Aspects, Legal Aspects, Data Security

Sector (JRC Taxonomy): Health, Energy, Financial, Government, etc.

Relation to emerging technologies: Artificial Intelligence, Big Data, Autonomous Machinery, Robotics

More details in the complete roadmap