Each week we find a new topic for our readers to learn about in our AI Education column.
Today’s artificial intelligence brings with it quite an interesting can of worms—especially when it comes to protecting information and ensuring that it provides the right answers and results to users.
There’s a huge technology topic that we’ve rarely—if ever—addressed so far in AI Education: Cybersecurity. Cybersecurity is a rapidly growing industry currently worth somewhere around $300 billion or so—yet we’ve barely touched it here it. This week, that’s going to change, because we’re going to talk about a type of cybersecurity breach that is unique to AI: The poisoned data attack, or data poisoning.
Data poisoning is pretty much unique to AI because AI does things that traditional computing does not do. That’s kind of what we mean by AI opening a can of worms as far a cybersecurity is concerned. For one thing, today’s AI often involves moving a lot of information to a lot of different places—along the way, that information has to be kept secure. Cybercriminals aren’t just operating online—hackers have a proud tradition of going so far as to dive into dumpsters to find passwords, account numbers, and even scraps of code that they might exploit—so moving data from place to place also requires layers of digital and physical security from place to place and in transit. If data is a money train, it has to be secure both in the station and on the tracks.
What Is Data Poisoning?
In data poisoning, actors seek to alter the data used to train artificial intelligence models to either break the models altogether, or influence them to produce skewed results.
Let’s back up for a second and recall that artificial intelligence models, like the ones powering AI chatbots, rely on accurate data not only to query an answer to our questions or generate an image for us, but also to learn how to read and converse in natural language. We’ve trained the models we use on huge sets of high-quality data, which has given them the knowledge and skill to work with our lower-quality queries in the inference stage, where we’re asking the technology to think, act and create on our behalf.
However, our current AI isn’t really able to account for biased and incorrect, or corrupted, malicious and broken data in the training stage—it’s a point of vulnerability. Bad data in training sets can be used to change the way a model behaves, or to make it cease to function entirely.
Types of Data Poisoning Attacks
IBM divides data poisoning into two overarching categories, targeted attacks versus non-targeted attacks, which can be extended to describe different classifications of cyberattacks in general, where poisoned data attacks fall into the category of adversarial attacks—any attack intended to fool AI or machine learning models to produce incorrect results or reveal sensitive information. A targeted attack seeks to change the output of a specific model or models in a specific way. A non-targeted attack seeks to degrade a model or models by impairing their ability to process data.
There are a myriad of different overlapping types and classifications of data poisoning attacks—some that might be encountered in the financial services industry include:
Availability Attacks—Data that slows model performance is injected, with an aim to slow or eventually crash the model.
Backdoor Poisoning—Data is injected that will cause the model, at a specific trigger known only to the attacker(s), to change behavior permanently or for a specific period of time. Backdoor attacks can remain latent—present but untriggered and unseen—for long periods of time before activated.
Clean-label Attacks—An attack that does not alter the way data is labeled, making many seemingly small changes within training sets difficult to detect by most data validation methods.
Gradient Descent Manipulation—Influences model behavior by subtly altering data over time without changing labels.
Label Flipping or Mislabeling—This attack, sometimes called a dirty-label attack, changes the way a model’s training data set labels pieces of data, leading it to misidentify certain items after training.
Data Injection—Data is added or altered within a model’s training data set, usually in a manner that biases the model in a specific direction.
Why Data Poisoning Is a Problem
Data poisoning connects to AI’s overall trust issues, because any AI model is still only as good as the information used to train it and the ability of users to query it. If the training data is in question, then we can’t trust—or use—the products of the AI, especially in highly regulated industries like financial services. As AI assumes more key functions and is used to process and manipulate more sensitive information, data poisoning of any type has a potential for catastrophe.
Think of the possibilities. A malicious hacker might use poisoned data in a targeted to influence malware and anti-virus software to miss certain vulnerabilities and threats, enabling them to gain access to more systems and sensitive data. In a non-targeted attack, malicious actors might alter computer vision so that electric vehicles could no longer sense pedestrians or read traffic signs accurately.
We haven’t even talked about the two areas where poisoned data could do the most damage: healthcare, where poisoned data could manipulate research, lead to suboptimal patient care and outcomes, or potentially kill someone, and finance, where poisoned data could be used to collect the information malicious actors need to rob someone blind—or potentially teach an AI model to do it autonomously. This is why AI comes with additional compliance and security concerns in these industries—our additional can of worms.
IBM lists a few of the impacts of data poisoning on AI models:
Reduced performance due to misclassification: Where poisoned data sets undermine the reliability of models, leading to inaccurate results and eroded trust in AI models.
Biased decision-making: Where data poisoning exacerbates existing biases in AI systems—which can be particularly devastating when said biases target specific demographic groups.
Security vulnerabilities: Where data poisoning opens the door for more malicious activity, including backdoor attacks and more sophisticated exploitations of the AI model.
