Research Note

When Fairness Metrics Disagree

Fairness evaluation in machine learning is often treated as a straightforward measurement problem. In practice, however, different fairness metrics can produce conflicting conclusions about the same model.

A system may appear fair according to one metric while showing substantial disparity according to another. This creates a major challenge for deployment decision-making in high-stakes AI systems.

Why Metric Disagreement Matters

Fairness metrics measure different forms of model behaviour. Some focus on false positive rates, others on false negative rates, demographic parity, or predictive consistency.

Because these metrics capture different properties, disagreement between them is not unusual. However, deployment decisions are often made without considering the uncertainty introduced by conflicting fairness evaluations.

Operational Risk

In high-stakes settings, fairness disagreement is not simply a reporting issue. It can become a deployment-risk issue.

Systems evaluated as acceptable under one metric may still introduce operational, ethical, or governance concerns when assessed under alternative fairness criteria.

Beyond Single-Metric Evaluation

Robust AI assurance increasingly requires:

• Multi-metric fairness evaluation
• Subgroup disparity analysis
• Threshold sensitivity assessment
• Context-aware deployment review
• Decision-risk analysis

Evaluating fairness through a single metric alone may provide an incomplete view of real deployment suitability.