Hacking 3.0: Data poisoning

Issue in brief

We may be seeing a sea change in hacking. Hacking 1.0 was tinkering with physical systems, some of which were electronic. That’s legacy hacking. Hacking 2.0 was writing malicious code or altering code in malicious ways. Often, this was aimed at having some effect on computer networks such as gaining remote control of systems or applications on endpoints.

Now, thanks to the prevalence of ML, information systems are sensitive to data. The fundamental hard problem of ML-based AI is that understanding the relationships between data inputs and the results they affect in areas such as search results or image classification results can remain opaque to ML-system developers. Those opaque dependencies lead ML-developers to rely on and put trust in large data sets. The functioning of ML-enabled information systems therefore depends on these data.

This sensitivity of information systems to data means that hacking 3.0 threat actors can focus their attacks on these very data themselves. They can manipulate data in such a way as to have real effects on information systems. This is not the traditional data manipulation threat objective, where the data resided in data bases and was often inert with respect to the functioning of the information system. ML-empowered information systems are now becoming problem-solving platforms. Now, the decisions computer systems make to perform core tasks can be altered by subverting input training data or toying with classifiers, and these decisions often have underlying rules that have the opacity problem. This means that even recorded events in the world and information about them are becoming a larger attack surface. These vulnerabilities have concerning implications for a range of social systems, from elections to welfare programs to college admissions.