Robust Replacement of Human Decision Makers under Distribution Shift
When should a machine learning algorithm replace a human decision maker when its deployment distribution may differ from the training data?
Read abstract
This paper studies when a machine learning algorithm should replace a human decision maker under distribution shift. We infer each human’s error rates from past decisions, assuming that they stay the same on future cases, while the data seen by the algorithm may shift within a Wasserstein ball. For each threshold, we compute the algorithm’s worst false positive and true positive rates over the ball, which gives a worst-case ROC curve. We replace a human only if some point on this curve is, with high posterior probability, at least as good as the human in both rates. We show that the estimated curve converges uniformly and bound the welfare loss from replacement. We also compare the rule with an infeasible benchmark that knows the deployment distribution and each human’s error rates and sets the algorithm’s false positive rate equal to the human’s. Relative to this benchmark, choosing the threshold with the highest posterior probability can leave a welfare loss that does not vanish with more data. In simulations, robust rules replace fewer humans than a rule that ignores shifts, giving up some gains without shifts but losing less after large shifts.