About AI Red Team

Abstract

The proliferation of machine learning applications based on large language models (ChatGPT, etc.) has brought attention to a well-known problem in machine learning systems: adversarial attacks. Such attacks are special modifications of data at different stages of the standard machine learning pipeline (training, testing, use), which are designed to either prevent the operation of machine learning systems or achieve the special behavior of such systems required by the attacker. In the latter case, the attacker usually wants to ensure that the trained model reacts in a special way (needed by the attacker) to input data prepared in a certain way. There are also classes of attacks on machine learning models that specifically interrogate running models in order to obtain hidden information used in training the model. All of the above attacks can be implemented quite simply for large language models, which opened the eyes of the business community to a real problem - the cybersecurity of machine learning (artificial intelligence) systems themselves. The answer was the accelerated creation of corporate cybersecurity units that should test artificial intelligence systems - AI Red Teams. The principles of the construction and operation of such teams are discussed in this article.