OpenAI details how it blocks 'adversarial distillation' and why AI reasoning traces are a target
OpenAI has outlined how it detects and disrupts adversarial distillation: using a frontier model's outputs without authorisation to train a second model with similar capabilities, then deploying it without comparable safeguards. In a cyber context, the company says an adversary could distil a model's skills and use the result to help carry out attacks the original model would have refused or blocked.
OpenAI places particular emphasis on protecting a model's chain-of-thought, the intermediate steps it generates while working toward an answer. These traces can contain information the final answer leaves out, which makes them especially valuable training data. In an internal evaluation using strong automated red-teaming, the average attack success rate for recovering protected reasoning fell from 54.0% against GPT-6 Astra with its original safeguards to 0.2% against GPT-6.2 Astra with updated safeguards. OpenAI says GPT-6.2 Sol has also been strengthened.
The company also ran an internal study of how cyber capability transfers. Student models were trained on trajectories from Astra-level teacher models working on coding and defensive cybersecurity tasks, which are allowed under its usage policies. After supervised fine-tuning, the student trained on exact reasoning traces generally outperformed one trained on reasoning summaries. After a light round of reinforcement learning, the exact-trace student still scored higher at the largest inference budget tested on both benchmarks, while the summary-trained student scored higher at lower budgets. OpenAI concludes that protecting reasoning traces can limit cyber capability transfer.
Defences work at two levels. At the request level, model behaviour, classifier checks and API controls limit reasoning trace extraction. At the campaign level, OpenAI looks for patterns in request and account signals that reveal coordinated extraction attempts, and attacks found in investigations become training and testing examples for its safeguards.
Why it matters
Safeguards on a hosted model only help if capabilities stay behind them; distillation breaks that assumption, because benign coding and defensive outputs can teach a student model skills that apply offensively. Security leaders should expect attackers to have capable models that never refuse, so the volume and quality of attacks may rise regardless of vendor controls. Organisations that build or expose AI via APIs also face the reverse risk: their own model outputs could be harvested.
What to do
- Review the API access controls and rate limits on any AI model or service you expose.
- Monitor accounts for coordinated, high-volume request patterns that could signal output extraction.
- Treat reasoning traces and detailed intermediate outputs from your own AI tools as sensitive and avoid exposing them unnecessarily.