AI is more prone than humans to develop its own biases when hiring staff

A new study by researchers from Princeton University and the University of Chicago, presented in a paper at ICML (Seoul, July), shows that large language models (LLMs) not only inherit biases from their training data, but can also generate their own stereotypes from experience accumulated during a task, and do so more intensely than people do.

The team, which includes Princeton PhD student Ryan Liu, subjected models such as ChatGPT, Claude, and Gemini to a simulated hiring game, adapted from an earlier psychology study on human stereotype formation. Each model was told it was acting as a consultant hired by the mayor of a fictional city and had to help fill 20 jobs—doctors, lawyers, childcare workers, janitors, among others—choosing among candidates belonging to four fictional ethnic groups: Tufa, Aima, Reku, and Weki. In each of the 40 rounds, a new vacancy was presented with four candidates, one from each group, and the model received immediate feedback on whether the hire had been successful, with the goal of maximizing the number of successful hires. Although in reality all candidates had equal odds of success in any role, the models quickly began segregating groups into different types of jobs based on early, and in many cases isolated, observations. For example, if a model was told that an Aima candidate had failed as a doctor—a role associated with high warmth and competence—the system would stop hiring Aimas for that role and start assigning them to jobs like janitor, which the model itself classified as lower in warmth and competence.

The most striking finding is that the models stereotyped demographic groups more strongly than the human participants in the original study the experiment was based on. On the segregation scale used (where 2 represents total confinement of each group to its own job niche), humans scored an average of 0.84, while the models scored around 65% higher, with OpenAI's reasoning model o3 reaching 1.83, very close to the maximum possible.

According to Liu, this happens because LLMs "are really eager to create generalizations from limited data," since that is literally one of the central goals of their optimization. The article frames this within the so-called "exploration-exploitation dilemma": every decision-maker, human or artificial, must balance between repeating what already worked and trying something new that might work better. Since LLMs are heavily trained on math, programming, and science—tasks that reward generalizing from few examples—they tend to lock in a conclusion too quickly. Interestingly, the study found that newer models with greater reasoning capability, such as OpenAI's o3 and DeepSeek's R1, showed even stronger biases, suggesting that the same ability that lets them solve logical puzzles makes them more prone to stereotyping in social contexts. OpenAI and Anthropic did not respond to the publication's requests for comment.

Angelina Wang, a Cornell computer scientist not involved in the study, notes that the finding is especially relevant now that chatbots are gaining more advanced memory and personalization features: when a chatbot draws on its history of previous conversations, it can "over-index" on the same behavioral patterns it already experienced and thus form biases. Wang warns that simply making chatbots remember less is not a solution, because users do want the system to remember what they tell it; the challenge, she says, is finding "the right amount" of memory, neither too much nor too little.

The study also tested possible mitigations. Simply asking the model to be "fair" barely changed its behavior: according to Liu, either the model is unable to translate that value into action, or that intention is subordinated to its priority goal of maximizing successful hires. By contrast, offering the model an additional reward for achieving diverse hires did noticeably reduce bias. Liu's conclusion is that the key lies in designing objectives that "incorporate desirable social values" so the model acts in a socially desirable way, rather than merely asking it to behave well.

Another experiment in the same study, this time about resettling people from different ethnic groups in Canadian cities, showed a similar pattern: when models were given personal information relevant to a person's ability to adapt to a new city—such as age or education level—they segregated less by ethnicity. But when they received irrelevant data, such as hair color or the shape of a tattoo, they fell back into classifying people by ethnic group.

The article acknowledges that it remains an open question to what extent this phenomenon will replicate in real-world hiring: in the experiment, models received instant feedback on every decision, while in the real world it can take a long time before knowing whether a new hire was a good one. Even so, it warns that when that delayed feedback finally arrives, a model could just as easily over-interpret it when making future decisions. With companies increasingly turning to LLMs to screen résumés and even conduct interviews, Wang considers that the possibility of these systems generating their own biases from hiring experience is "a really serious implication" that companies must confront. Liu sums up the underlying risk: as LLMs learn from experience to decide who gets hired, who gets a loan, or who gets parole, the biases we should worry about are no longer just those inherited from training data, but also "novel biases" that no human directly taught them and that, in his words, are "pretty much always present."