Apple Card offered vastly different limits to spouses with identical assets.
David Heinemeier Hansson was offered a credit limit 20 times higher than his wife despite having the same shared assets. The algorithm likely learned from a world where men earn on average more than women and concluded that men are more creditworthy. This shows how bias creeps in even when protected characteristics like gender aren’t direct features. The training data itself was biased, reflecting existing societal inequalities, and the algorithm learned to perpetuate them without explicitly considering gender.
— It’s The Data, Stupid! Why AI Might Get It Wrong. · Forbes