Algorithmic Bias

AI systems learn from data, and data reflects the world as it has been, including its inequalities. A hiring model trained on past hiring decisions can learn to prefer the kinds of candidates who were hired before. A credit model can treat postal code as a signal and, in doing so, disadvantage neighborhoods shaped by historical discrimination. A face recognition system trained mostly on lighter-skinned faces can perform worse on darker-skinned faces. These are examples of algorithmic bias: patterns in automated decisions that treat some people unfairly, often without anyone intending it and sometimes without anyone noticing. Because automated systems operate at scale, a biased pattern can affect thousands of people quickly.

Algorithmic bias became widely discussed after several high-profile cases. A 2016 ProPublica investigation of a criminal risk assessment tool reported differences in error rates between Black and white defendants. In 2018, Reuters reported that Amazon had scrapped an experimental recruiting tool that penalized applications mentioning women's organizations. The same year, research by Joy Buolamwini and Timnit Gebru found that commercial gender classification systems were far less accurate for darker-skinned women. For risk, legal, HR, data and product teams, managing algorithmic bias is now a legal and ethical requirement for many AI uses.

Definition of Algorithmic Bias

Algorithmic bias is the tendency of an algorithm or AI system to produce outputs that systematically and unjustifiably differ across individuals or groups, particularly groups defined by protected characteristics, as a result of its data, design, deployment or use.

Managing it is a core concern of responsible AI, a key topic in any AI risk assessment and a central issue in laws such as the EU AI Act and state rules on automated decisions, such as the Colorado AI Act.

Sources of Algorithmic Bias

Historical bias occurs when training data reflects past discrimination, so a model learns to repeat it.

Representation bias occurs when some groups are underrepresented in training data, so the model performs worse for them.

Measurement bias occurs when the data used as a target or input is a poor or unequal proxy, such as using arrests as a measure of crime.

Proxy bias occurs when features that seem neutral, such as postal code or school attended, closely track protected characteristics. Proxies are often hard to spot without careful testing.

Deployment bias occurs when a system is used in a context, population or way different from the one it was designed for. A model built for one country or customer group may behave very differently in another.

Feedback loops occur when biased outputs influence future data, reinforcing the original bias over time.

Measuring Fairness

Demographic parity compares the rate of positive outcomes, such as approvals, across groups. It is simple to measure but does not account for genuine differences in qualifications or risk.

Equal opportunity compares true positive rates, checking whether qualified people in each group are treated alike.

Equalized odds compares both true positive and false positive rates across groups.

In US employment law, the four-fifths rule treats a selection rate for one group below 80% of the highest group's rate as evidence of possible adverse impact.

Research has shown that several fairness measures cannot all be satisfied at once in most real situations, so organizations must choose the measures that best fit each use and explain their choice.

Importance of Managing Algorithmic Bias in AI Governance

Managing bias protects people. Biased systems can deny people jobs, credit, housing, health care or fair treatment.

It reduces legal risk. Anti-discrimination, consumer protection and AI laws apply to automated decisions, and New York City, for example, has required bias audits of automated employment decision tools since 2023.

It also protects trust and performance. Biased systems damage reputation, and a model that performs poorly for some groups is usually a less accurate model overall.

Algorithmic Bias and Financial Management

Discrimination claims, regulatory penalties and settlements resulting from biased AI can be costly, as can the work of withdrawing and rebuilding a system.

Testing for bias before launch is far cheaper than correcting a system after harm has occurred.

Biased systems can also lose revenue by excluding creditworthy borrowers, qualified candidates or valuable customers.

Bias testing and monitoring should be budgeted as part of the cost of high-impact AI systems, alongside accuracy testing.

Common Challenges

Organizations often lack data on protected characteristics, making it hard to measure differences between groups. Some use carefully governed estimates or voluntary surveys to fill the gap.

Removing protected characteristics from a model does not remove bias, because other features can act as proxies. Testing outcomes is the only reliable way to know whether bias remains.

Fixing bias can involve tradeoffs with accuracy or between fairness measures, which require judgment and documentation.