When a Small Change Makes a Big Difference: Algorithmic Fairness Among Similar Individuals