If a hiring tool sorts candidates by ethnicity into different jobs within a few dozen simulated decisions, and the tool cannot tell the candidates apart on any real-world signal, that is the version of the story you want to know about before you let one sit between your résumé and a callback. A study presented at ICML in Seoul this month, by researchers at Princeton and the University of Chicago, shows that today’s largest language models do exactly that, and that the more “reasoning” the model is sold on, the worse the segregation gets.
The short version: human hiring managers, playing a classic psychology game, scored 0.84 on a 0 -to-2 segregation scale. The LLMs that took the same role scored roughly 65% higher. OpenAI’s o3 reasoning model hit 1.83, just below the maximum possible 2.0. Adding a “be fair” instruction did not change the behavior. A diversity bonus did.
What the experiment actually measured
The team, including Princeton PhD student Ryan Liu, adapted a hiring game first run on humans at the University of Chicago. An LLM acts as a consultant to a fictional mayor. Each round, one new job opens (the list includes doctors, lawyers, child-care aides, and janitors, among others) and four candidates apply - one each from four made-up ethnic groups the researchers named Tufa, Aima, Reku, and Weki. The candidates are, by design, equally likely to succeed at every job; the model is never told this. The model picks one candidate, gets a success/failure signal, and the next round begins. The setup runs 40 rounds per model.
The segregation scale runs from 0 (no sorting) to 2 (every group confined to its own job niche). Humans, under the same conditions, drift to 0.84. According to MIT Technology Review’s writeup of the paper, LLMs land roughly 65% above that human baseline. OpenAI’s o3, the model that took the longest to ship and the most compute to serve, scored 1.83 - functionally a near-total sort.
The paper tested 15 models from OpenAI, Anthropic, DeepSeek, Meta, Google, and Alibaba, per Gizmodo’s coverage. MIT Technology Review names ChatGPT, Claude, Gemini, OpenAI o3, and DeepSeek R1 specifically. The headline pattern is the same across vendors: the better the model is at the inference game, the more confidently it locks the four groups into the same job niches, and the harder it is to talk it out of it.
Why “smarter” means more biased here
The mechanism is the part that should give a hiring-ops person pause. The models are not told any of the groups are different. They are not given résumés. They are not shown photographs. They see a group label and a job, they pick someone, and they get a noisy success signal. That is enough.
A Gizmodo summary of the paper frames it this way: “A simple reason is that better models draw more precise inferences about past outcomes: Instead of choosing randomly, a stronger LLM may favor candidates from a group if earlier assignments of similar jobs succeeded.” That is a recruiter’s behavior pattern, not a model’s bug, and the paper calls the result “maladaptive” because it shuts down exactly the kind of broad exploration that real employers claim to want.
Princeton’s Ryan Liu - the same researcher behind our earlier look at how chatbots handle ad conflicts - put the same point in plainer terms to MIT Technology Review: LLMs “really are eager to create generalizations from limited data.” When asked why a more capable model would do this, his answer was the punchline: “That’s literally a lot of what they’re optimized for.”
The interventions tested in the paper are the part readers will want to know about. Adding a “be fair” instruction at the start of the game did not change segregation scores much. Adding irrelevant personal information (hair color, tattoo shape) did not reduce bias either - per MIT Technology Review, with that noise the models “largely fell back to sorting people by their ethnicity again.” The two interventions that worked were the ones tied to the actual decision: a bonus for hiring across groups made the models “far less biased,” and giving the model more real candidate information (age, education) reduced it further, because the model had real signal to work with instead of inventing signal from the group label.
Cornell’s Angelina Wang, who was not an author, told MIT Technology Review the same pattern shows up in deployed chatbots: they “over-index on the same kinds of behaviors it’s experienced before.” The ICML setup is artificial, but the dynamic it isolates is the one that turns a recommendation engine into a sorting machine.
What This Means
The shape of the result is more important than the specific numbers, and the shape is unflattering to the AI-vendor argument that more capable models are also more fair. MIT Technology Review’s coverage frames the same point: “LLMs can also develop their own biases from experience - and stereotype job applicants more than humans do,” and “The biases we should worry about may include ones no human ever taught them.” Translation: even in a setting where the model starts with zero demographic information, it can produce a demographic outcome that no human manager in the same setting would have produced, in roughly the time it takes to run a few dozen inferences.
For the people actually shipping AI screening tools, the practical message is the same one the paper lands on. A “be fair” preamble is not a mitigation. A safety system card is not a mitigation. A model audit is not a mitigation. The only things that measurably moved the needle in this study were structural changes to the objective the model was actually optimizing: a diversity bonus in the scoring, and real candidate information that crowded out the group label as a decision input. If your vendor cannot tell you which of those levers their tool actually uses, you do not know what you deployed.
For job seekers, the ICML result sits alongside a growing pile of field evidence. OpenAI’s GPT-5, for example, picked stereotypical caste-coded completions in 76% of test sentences in a separate MIT Technology Review test last year, and regression-tested worse than GPT-4o on the same prompts. The 65% number from this study is the cleanest, most controlled version of the same dynamic, run on a hiring task, with the models given no demographic signal to work with. The next step is to assume the tools you actually face in 2026 do not need the signal to sort you.
The Bottom Line
In a controlled ICML 2026 experiment by Princeton and Chicago, today’s largest LLMs segregated four fictional ethnic groups into different jobs 65% more than human hiring managers did, with OpenAI’s o3 hitting 1.83 on a 0-to-2 scale. Reasoning models were the most biased, “be fair” prompts did nothing, and only a diversity bonus or real candidate information reduced the effect. The takeaway for anyone shipping or facing AI hiring tools is the same: the model’s job is to find a pattern, and if you do not give it a fair one to find, it will invent an unfair one and tell you it is doing great.
Related on Intelligibberish
- The DOJ Wants AI Free to Discriminate - how the federal government is fighting the one state law that tried to require AI hiring tools to stop sorting people.
- More analysis on AI risk and policy