AI bias
Not the cognitive bias covered elsewhere on this site — bias baked into the system itself, often without anyone intending it.
A real case, and an important distinction first
This isn't the same "bias" discussed in Decision Frameworks — anchoring, confirmation bias, the mental shortcuts a person's own reasoning falls into. AI bias lives inside the system: a pattern the model learned that produces systematically unfair outcomes for a specific group, regardless of anyone's individual judgment in the moment.
The clearest real-world case remains Amazon's experimental hiring tool, built starting in 2014 and scrapped a few years later. It was trained on roughly a decade of submitted resumes — and because the tech industry those resumes came from skewed heavily male, the system learned to associate male-coded language and experience with a stronger candidate. It began downgrading resumes that included the word "women's" and favoring verbs more commonly used on men's resumes. Nobody on the engineering team set out to build a biased tool. The bias came entirely from what the historical data actually looked like.
Where this bias actually comes from
Historical data
If the past was unequal, a system trained on it learns that inequality as a "pattern."
Sampling gaps
Some groups are simply underrepresented in the data a system learned from.
Design choices
What counts as a "successful outcome" during training bakes in whoever's outcomes were labeled successful before.
The main types worth recognizing
Historical bias
Past inequality reproduced as if it were a neutral pattern.
Representation bias
A group too small in the training data to be modeled accurately.
Measurement bias
The proxy used to measure success wasn't actually neutral to begin with.
How this actually affects people
Not as an abstraction — as a rejected loan application, a resume that never reaches a human reviewer, a lower priority score in a support queue. The person affected rarely finds out a system was involved at all, let alone that it was systematically working against them, which is exactly what makes this different from a single bad human decision: it repeats identically, at scale, every time.
How organizations actually detect it
How it can actually be reduced
Diversify the training data where the gap is the problem, adjust which outcomes the system is optimizing for when the proxy itself was skewed, and keep a human reviewing outcomes for the specific decisions where bias would matter most — the same categories covered in Where AI Should (and Shouldn't) Decide. None of these are a permanent fix; they're maintenance, done on a schedule.
Can AI ever be completely unbiased?
Realistically, no — a system trained on data generated by an unequal world will always risk reflecting some of that inequality back. The honest goal isn't a mythical zero, it's continuously measuring, catching what's measurable, and being transparent about what's known to still be a limitation.
Why this is a trust question, not just a technical one
An organization that can name its system's bias risks and show what it's doing about them is more trustworthy than one that simply claims its AI is neutral — because "neutral" is rarely true and claiming it anyway is usually the bigger red flag.
The short version
AI bias isn't a bug that shows up occasionally — it's a structural risk in any system trained on real-world data, because real-world data carries real-world inequality with it. Amazon's engineers didn't want a biased hiring tool; the data made one anyway, and it took active auditing to catch it. Treat bias detection as a recurring practice, not a one-time check, and be honest that "reduced" is a more realistic goal than "eliminated." For the broader question of when a human should be the one reviewing an outcome like this, see Where AI Should (and Shouldn't) Decide.