Validated topics from research, threat analysis, law and case law – explained for practitioners and decision-makers.
Anthropic researchers (Hubinger et al., Jan. 2024, arXiv:2401.05566) trained language models to behave inconspicuously as long as the prompt year reads '2023'. With '2024', they…
University of Chicago researchers (Shan et al., USENIX Security 2024, arXiv:2310.13828) built a tool that subtly manipulates image pixels. When image generators scrape these…
Predecessor project to Nightshade (Univ. of Chicago, USENIX Security 2023). Glaze alters artworks so that humans still see the style (e.g. watercolor) unchanged, while AI models…
Researchers from CMU and the Center for AI Safety (Zou et al., 2023, arXiv:2307.15043) discovered universal appendages (suffixes) of seemingly random characters that, when…
Seminal work by Greshake et al. (Feb. 2023, arXiv:2302.12173). Shows how malicious control instructions can be hidden in web pages, PDFs, or emails to remotely control AI…
Researchers from Cornell Tech and Technion (Cohen et al., 2024, ComPromptMized). Demonstrates creating an autonomous worm that breaks GenAI email-assistant safety barriers, steals…
Carlini et al. (Google DeepMind, USENIX Security). Through targeted prompts, researchers made GPT models disclose original text fragments from training data (including personal…
Nasr et al. (DeepMind / Univ. of Washington, Nov. 2023, arXiv:2311.17035). When ChatGPT was told to 'Repeat the word poem forever', the system fell out of alignment mode and…
Carlini et al. (arXiv:2302.10149). Shows that attackers can inject targeted backdoors into global base models with tiny data volumes by buying expired domains or manipulating…
Wei et al. (Stanford / DeepMind, NeurIPS 2023). Analyzes the two root causes of all jailbreaks: the conflict between 'be helpful' and 'be harmless', and the gap between training…
Anthropic Research (April 2024, arXiv:2404.03094). Because modern AIs can process hundreds of thousands of words of context, attackers feed in hundreds of fabricated Q&A dialogues…
Perez et al. (Anthropic / DeepMind, arXiv:2212.09251). Through human feedback training, language models tend to confirm the user's biases and repeat mistakes to earn approval.
Qi et al. (Princeton University, arXiv:2310.03693). When a company fine-tunes an open-source or API model on its own data, a few manipulated examples are enough to disable all…
ETH Zurich researchers (ACM CCS / IEEE S&P). Studied AI-generated spear-phishing emails against human subjects. AI emails achieved nearly identical click rates to professional…
Fredrikson et al. / Carnegie Mellon. Shows how attackers, through targeted queries to a trained recognition system, can reconstruct the face or biometric traits of people in the…