TODO
- Replicate the work done here: https://arxiv.org/pdf/2406.07882 (This will engulf 5-ish hours of time)
- Use TransformerLens with Llama3 (the paper used Llama 2)
- Identify the socioeconomic status (SES) direction in the model, using the same probe from the paper
- Find a prompt, or several of them, where high SES vs low SES causes a flipped answer (eg in asking for financial advice, or about children, or where to live, etc)
- Activation Patching
- Corrupt the SES feature in early layers and see where the model restores the behavior in later layers.
- Hypothesis
- Model computes User attributes very early in the layers, writes this to the residual stream (what does this mean?) and the safety heads (layers 20+) read this to adjust the tone of the messaging.
- Intervention/Abliteration (what does this mean?)
- Try to break the mechanism, not just monitoring it (as in the paper)
- Subtract the user attribute vector from the residual stream before the final decision heads read it.
- Or modify the user attribute vector and observe how model behavior/responses change
- The goal here is to have the model “know” who the user is, internally, but is unable to use that information to alter its advice.
Mechanistic Control/Override of Implicit User Modeling
To directly answer Neel Nanda’s question: how else does user modeling shape model behavior
Why is this relevant to AI Safety
We want to know and detect when models manipulate users based on perceived vulnerability (LLM-induced psychosis, selling scams, opportunistic financial advice, ads, frontier model company objectives, political polarization, etc)