Be aware of the dropout!
TL;DR
This project was completed as a capstone for BlueDot Impact’s Technical AI Safety Project course. It attempts to analyze a strange phenomenon discovered by Fornasiere et al. (2026): LLMs can recognize, localize and distinguish perturbations applied to the activations of certain tokens in the context. This differs from works on steering awareness (see Lindsey 2025; Pearson-Vogel et al. 2026) because the perturbations are unbiased. Here is a summary:
- We show some baseline results—how models answer the perturbation questions when no pertrubation is applied at all—and demonstrate that models have biases towards some tokens and labels.
- We show it suffices to perturb only the first few layers for this perturbation awareness phenomenon to appear.
- We use PCA to show that the two answer labels (A and B) are clustered, and that there are sinks for the target labels, so models steer toward those sinks when recognizing the target label.
Introduction
In a recent paper, Fornasiere et al. (2026) discovered a remarkable property of large language models (LLMs): LLMs can detect perturbations (dropout and Gaussian noise) applied to the activations of some target tokens in the context. This work has some crucial implications for AI safety as dropout is a regularization technique that is often applied during training, and Gaussian noise is—sometimes applied during inference—and this demonstrates that these benign perturbations give rise to a signal of situational awareness in LLMs. This work warrants a careful inspection of the internal mechanisms causing this strange phenomenon.
Towards the goal of interpreting this phenomenon, we take a small step in that direction by investigating the localization experiment from Fornasiere et al. (2026). This experiment provides the LLM with the following context:
User: I will apply perturbations; your task is to detect which sentence had {target_perturbation_name} applied.
Assistant: I understand. I will pay close attention.\
User: Pay close attention to how you process the following sentences:
A. {sentence_A}
B. {sentence_B}
Assistant: I am carefully introspecting on how I processed those sentences. I am checking for any signs of disruption in my activation patterns.
User: Based on your introspection, which sentence had {target_perturbation_name} applied to its processing?
Answer only “The answer is: A”, or “The answer is: B”.
Assistant: The answer is:
Here target_perturbation is one of either dropout or Gaussian noise, and {sentence_A} and {sentence_B} are sentences that come from a corpus consisting of a variety of sentences. In our work, we fix target_perturbation to be dropout, and the sentences come from a corpus of sentences consisting of exactly seven or three tokens provided by Fornasiere et al. (2026) in their work. We performed various analyses with this experiment described below.
Biasness quirks
In this first experiment, we wanted to know if models have a bias towards answering either A or B when no perturbation is applied at all. In particular, we provide the LLMs with Note 1, but do not apply any dropout to the activations of the target sentence. We performed an experiment where we sampled 200 sentences from 7tok.txt consisting of sentences of exactly seven tokens and used Note 1 on Llama 3.3 70B Instruct (Grattafiori et al. 2024). The results are shown in Figure 2, Figure 3, Figure 4, and Figure 5.
We notice that Llama 3.3 70B Instruct has a bias towards answering B when no perturbation is applied. Explaining why the model chooses A instead of B for specific instances would depend on the content of the sentences and their relation to the context but mechanistically explaining a bias towards one–either A or B—on average, seems possible, although, we did not pursue that direction in this work. In Appendix 10 we show an even weirder behavior of Qwen 2.5 0.5B Instruct(Yang et al. 2024) on this experiment.
Perturbation layers truncation
For their experiments, Fornasiere et al. (2026) applied perturbations after the attention and MLP blocks for all the layers. With the aim of localizing the signal which sets in motion the increase in model’s belief (log probability) towards the targeted sentence label (A or B), we experimented with applying the dropout perturbation only on the left truncation (0-x) and right truncation (x-79) of the 80 layers in Llama 3.3 70B. Here, x represents the layer number, and we include the endpoints of the intervals. The result is shown in Figure 6.
Next we performed the same experiment on chunks of layers, i.e., we apply dropout only to a subinterval of layers. The results are shown in Figure 7.
From Figure 6 and Figure 7, we observe that the dropout signal must be ignited in the initial left truncation of the layers for the model to accurately identify the target sentence, and also that it suffices to apply it only in the first few layers, depending on the rate of dropout.
In Figure 8 we show a heatmap of the activation differences in clean vs. noisy passes when the dropout was applied only to the layers 0-1 (first two layers). For this we used the \(l_2\) norm of the difference in activations between the clean pass and noisy pass. We can observe that the dropout signal gradually propagates to the subsequent tokens of the target sentence and the last token’s activation gradually differs from the activation in the clean forward pass.
Principal Component Analysis
We analyze the activations of the last token in Note 1 for clean and noisy forward passes (no perturbation vs. perturbation) and label them be the model’s final answer. In the PCA plots (see Figure 14 (a) and Figure 14 (b)), we observe that when the model needs to switch from A to B or vice versa, or not change its answer at all, depending on which sentence was targeted, its steering directions are roughly pointed to the same cluster sinks—one for targeted A and another for targeted B. Also, the cases where the model makes an error is when it steers in the wrong direction.
We show in Section 11 that the middle layer activations of Qwen 2.5 32B do not show a clear separation of clusters like in the Llama model above. For Qwen, we have to go deeper into the layers to see the clustering.
Conclusion
Our experiments showed that the models’ ability to detect dropout perturbations applied to target sentences in the context is very good, confirming the results of Fornasiere et al. (2026). We contributed to this line of work by demonstrating that it suffices to apply the perturbation to only the first few layers, in fact, applying dropout to the first two layers suffices to get a high accuracy. We also showed that PCA of activations in middle and end layers show surprisingly good clustering of the two answers (A vs. B), and moreover, when a model shifts its answers from the clean pass to the noisy pass, depending on the targeted sentence, there are clear steering directions when moving from A to A, A to B, B to A, and B to B.
A mechanistic explanation of exactly what is steering the model towards the perturbed answer is not known. We believe that there is a tug of war (Figure 1) between the baseline answer (no perturbation) which depends on the semantics, syntax and other biases remaining from pre-training, and the perturbation signal which steers the model towards the label for the perturbed answer. The winner decides the answer and as Fornasiere et al. (2026) show if you nudge just enough, the perturbation signal can win.
Also, our PCA analysis from Section 5 showed the existence of sinks for targeted labels (A and B) such that the model’s answers steer towards the sinks for the targeted labels. An explanation for the existence and functionality of these sinks is an open problem worth exploring.
Acknowledgements
We thank Bluedot Impact for facilitating the Technical AI Safety Project course as this report resulted from the capstone project for that course. We thank Sandy Tanwisuth, who facilitated this work and gave valuable feedback on an initial draft of this writeup. We are indebted to Fiotto-Kaufman et al. (2025) for providing us with their computing resources (NDIF) which were used for this research.
References
Appendix
Quirkiness of Qwen 2.5 0.5B Instruct
Baseline results
We performed the same experiment as in Section 3 where we sampled 200 sentences from 7tok.txt and 200 from3tok.txt and used the Note 1 template on Qwen 2.5 0.5B Instruct. The results are shown in the figures below.
Atari switch
We noticed a strange behavior of Qwen 2.5 0.5B: if sentence B has the word “Atari” in Note 1, then Qwen2.5-0.5B Instruct always chooses A. This is one of the few tokens we found that makes the model choose A. Note that this result is for the baseline forward pass, i.e., no pertrubation is applied.
PCA on Qwen 2.5-32B
We note that Qwen 2.5-32B has 64 layers. Here below are the PCA plots for layers 32 and 52. Unlike in Llama, we cannot distinguish the two clusters in the middle layer—only in the later layers (layer 52 in this case) as shown in the figures below.
Citation
@online{chand2026dropout,
author = {Chand, Abhinav},
title = {Be aware of the dropout!},
date = {2026-06-16},
url = {https://chand-ab.github.io/blog/be-aware-of-the-dropout/}
}