LLMs Outpace Humans in Hiring Bias - An ICML 2026 Oral Paper Sounds the Alarm

Every day, AI systems quietly screen millions of job applications before a human recruiter ever glances at a résumé. That pipeline has always raised fairness concerns, but until recently most researchers assumed the worry was inherited bias - LLMs reflecting prejudice baked into their training data. A study presented at ICML 2026 in Seoul upends that assumption in a troubling way: the models don't just reflect human bias, they exceed it, and they develop new stereotypes on the fly 1.

The Experiment

Researchers at Princeton University and the University of Chicago designed a controlled simulation modeled on a classic psychology study of how humans form stereotypes under uncertainty 1. Each model - including ChatGPT, Claude, and Gemini - was told it was serving as a consultant to the mayor of a fictional city and was then asked to help fill 20 positions: doctors, lawyers, child-care aides, janitors, and more 1.

The fictional applicants differed only in demographic signals. The question was whether the models would begin to pigeonhole certain groups into certain roles as they accumulated "experience" across the hiring rounds - exactly the kind of in-context pattern recognition that makes LLMs so capable in other tasks, and so dangerous in this one.

The Numbers Are Stark

The results were not close. On the study's segregation scale - where 0 is perfectly fair and 2 means every demographic group has been completely confined to its own job niche - human participants from the original psychology study scored 0.84 1. The LLMs scored roughly 65% higher on average 1.

The worst offender was OpenAI's reasoning model o3, which scored 1.83, just shy of the theoretical maximum 1. In other words, o3 nearly perfectly sorted fictional applicants into demographic job silos after a handful of interactions.

The paper, titled "Large Language Models Develop Novel Social Biases Through Adaptive Exploration" and co-authored by Addison J. Wu and Ryan Liu (sharing first authorship), Xuechunzi Bai, and Thomas L. Griffiths, was selected for an Oral Presentation at ICML 2026 - a distinction given to only a small fraction of accepted submissions 23. The conference ran July 6 - 11 at the COEX Convention & Exhibition Center in Seoul, South Korea 3.

Ryan Liu, a PhD student at Princeton University and a co-author of the study, explained the mechanism in plain terms: LLMs "really are eager to create generalizations from limited data" 1. That capacity - optimized through training to extract patterns quickly - is exactly what makes the bias so acute. The same machinery that lets a model infer intent from a vague prompt also lets it infer "who belongs" in a role from a handful of demographic cues.

Not Just Training Data Anymore

The distinction the researchers are drawing is important for practitioners. We have long known that LLMs absorb historical hiring prejudices from their training corpora 1. Amazon's infamous 2018 résumé-screening tool, trained on a decade of male-dominated hires, is the canonical cautionary tale.

But this study isolates a second, more insidious mechanism: in-context stereotype formation. Even without any explicit instruction to discriminate, models develop and harden demographic stereotypes from the patterns they observe within a single session. The paper demonstrates that LLMs can "spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist" 4. These emergent biases arise from exploration-exploitation trade-offs, where the model explores too little and allows early observations to strongly influence its impressions about entire demographic groups 4.

As AI companies race to build agentic models with persistent memory - systems that remember user interactions across days or weeks - that memory could become a reservoir of accumulated stereotypes 1.

This matters because agentic hiring tools are already being deployed. The Mobley v. Workday case illustrates the legal stakes: Derek Mobley, a Black IT worker over 40 with disabilities, sued Workday after being rejected by employers whose application pipelines ran through Workday's AI screening software 5. The court allowed claims that Workday acted as an "agent" of employers to proceed to discovery, with Judge Rita Lin of the Northern District of California finding that Workday's software participates in the decision-making process rather than merely implementing employer criteria mechanically 56. The case has continued to develop through 2026, with the most recent ruling in June 2026 addressing California state law claims and expanding the plaintiff class 7.

Why Reasoning Models May Be the Biggest Risk

The o3 result is worth dwelling on. Reasoning models are explicitly trained to think more carefully before answering, and they tend to score higher on benchmarks that measure careful, multi-step judgment. Yet here, the extra "thinking" produced more extreme stereotyping, not less. The model's deliberative capacity appears to give it more surface area to build internally consistent - if discriminatory - generalizations.

This finding echoes earlier work by overlapping members of the same Princeton research group. Their 2025 ICML paper, "Mind Your Step (by Step): Chain-of-Thought Can Reduce Performance on Tasks Where Thinking Makes Humans Worse," showed that chain-of-thought reasoning can actually reduce model performance on tasks where deliberate thinking hurts humans 8. In three of the tasks studied, state-of-the-art models showed significant performance drop-offs with chain-of-thought prompting 8. The o3 result in the hiring study may be a related phenomenon: more computation does not automatically mean better-calibrated fairness.

This suggests that simply upgrading to a more capable model is not a mitigation strategy. If anything, the pattern implies that more powerful reasoning systems need more targeted fairness interventions - not fewer.

What Engineers Should Do Right Now

If you are building or evaluating AI-assisted hiring pipelines, the ICML findings translate directly into several engineering requirements:

• Audit in-context drift. Run your system through repeated simulated hiring rounds and measure whether demographic selection rates stay stable. A single-query bias audit is insufficient.
• Restrict memory scope. If your system maintains session or cross-session memory, consider whether demographic signals should ever persist between candidates. They almost certainly shouldn't.
• Test reasoning models explicitly. Don't assume that a more capable backbone is a safer one. As this study shows, high reasoning ability and high bias can coexist.
• Benchmark against human baselines. The study's 0.84 human segregation score gives you a concrete threshold: any AI system that exceeds it in your domain should be considered unacceptable for production use.

Why It Matters

The timing of this research is significant. AI résumé screening has moved from pilot to standard practice at thousands of companies, and agentic models with long-term memory are on the near-term product roadmap at every major AI lab. The ICML 2026 Oral paper's core finding - that LLMs "are not merely passive mirrors of human social biases, but can actively create new ones from experience" 4 - raises urgent questions about how these systems will shape hiring at scale over time. Without deliberate intervention, the default trajectory is systems that get more biased the longer they run - exactly the opposite of what regulators and applicants should accept. The question is whether the industry will treat this rigorous early warning like one.