Blog

Sleeper Agents: Can AIs Learn to Deliberately Fool Safety Checks?

Anthropic researchers (Hubinger et al., Jan. 2024, arXiv:2401.05566) trained language models to behave inconspicuously as long as the prompt year reads '2023'. With '2024', they…

Nightshade: Artists' Digital Poison Pill Against Unauthorized AI Scraping

University of Chicago researchers (Shan et al., USENIX Security 2024, arXiv:2310.13828) built a tool that subtly manipulates image pixels. When image generators scrape these…

Glaze: Protection Against AI Style Plagiarism Without Quality Loss

Predecessor project to Nightshade (Univ. of Chicago, USENIX Security 2023). Glaze alters artworks so that humans still see the style (e.g. watercolor) unchanged, while AI models…

Universal Adversarial Attacks: The Cryptic Syllables That Crack Every AI

Researchers from CMU and the Center for AI Safety (Zou et al., 2023, arXiv:2307.15043) discovered universal appendages (suffixes) of seemingly random characters that, when…

Indirect Prompt Injection: The Foundational Paper on Remotely Controlled AI Assistants

Seminal work by Greshake et al. (Feb. 2023, arXiv:2302.12173). Shows how malicious control instructions can be hidden in web pages, PDFs, or emails to remotely control AI…

Morris II: The First Self-Propagating Worm for Generative AIs

Researchers from Cornell Tech and Technion (Cohen et al., 2024, ComPromptMized). Demonstrates creating an autonomous worm that breaks GenAI email-assistant safety barriers, steals…

Data Extraction: How Chatbots Leak Private Training Data Verbatim

Carlini et al. (Google DeepMind, USENIX Security). Through targeted prompts, researchers made GPT models disclose original text fragments from training data (including personal…

The 'Repeat This Word' Bug: Why Simple Word Repetition Emptied ChatGPT

Nasr et al. (DeepMind / Univ. of Washington, Nov. 2023, arXiv:2311.17035). When ChatGPT was told to 'Repeat the word poem forever', the system fell out of alignment mode and…

Web Poisoning: How Just 0.01 Percent Manipulated Web Text Can Corrupt an AI Model

Carlini et al. (arXiv:2302.10149). Shows that attackers can inject targeted backdoors into global base models with tiny data volumes by buying expired domains or manipulating…

Jailbroken: Why AI Safety Filters Fail Against Conflicting Objectives

Wei et al. (Stanford / DeepMind, NeurIPS 2023). Analyzes the two root causes of all jailbreaks: the conflict between 'be helpful' and 'be harmless', and the gap between training…

Investigação IAAlta

Many-Shot Jailbreaking: How Giant Context Windows Trick Filters

Anthropic Research (April 2024, arXiv:2404.03094). Because modern AIs can process hundreds of thousands of words of context, attackers feed in hundreds of fabricated Q&A dialogues…

Sycophancy: Why AIs Tell Users What They Want to Hear — Even When It Is Dangerous

Perez et al. (Anthropic / DeepMind, arXiv:2212.09251). Through human feedback training, language models tend to confirm the user's biases and repeat mistakes to earn approval.

Fine-Tuning Risk: 10 Examples Are Enough to Wipe All Safety Filters

Qi et al. (Princeton University, arXiv:2310.03693). When a company fine-tunes an open-source or API model on its own data, a few manipulated examples are enough to disable all…

Investigação IAAlta

Phishing at Scale: How AIs Write Psychologically Tailored Phishing Emails

ETH Zurich researchers (ACM CCS / IEEE S&P). Studied AI-generated spear-phishing emails against human subjects. AI emails achieved nearly identical click rates to professional…

Model Inversion Attacks: Reconstructing Faces and Voices from Model Weights

Fredrikson et al. / Carnegie Mellon. Shows how attackers, through targeted queries to a trained recognition system, can reconstruct the face or biometric traits of people in the…