LLMs can exhibit stereotypical biases, which leads to harm, and it is crucial to avoid this issue. This paper aims at uncovering the source of these biases by using several adversarial attacks techniques: gradient based attack, jailbreak prompting and model red-teaming. These techniques successfully increase the number of stereotypical answers by up to 34% (51% for stereotypes against Black people) while maintaining excellent general performance. However, fine-tuning a model on stereotypes remains more efficient to increase the number of harmful answers, underlying the importance of training data. We make our datasets and code publicly available for better reproducibility.