A/B testing can't tell you why a change is better
I think this is a good example of the is-ought problem in philosophy, transplanted into the world of software development:
A/B testing is a great way of finding out what happens when you introduce a change. But it can’t tell you why.
I used the site earlier in the year, and actually felt my heart rate increase. Never again.
— Paul Robert Lloyd (@paulrobertlloyd) October 24, 2017
The problem is that, in a data-driven environment, decisions ultimately come down to whether something works or not. But just because something works, doesn’t mean it’s a good thing.
If I were trying to convince you to buy a product, or use a service, one way I could accomplish that would be to literally put a gun to your head. It would work. Except it’s not exactly a good solution, is it? But if we were to judge by the numbers (100% of people threatened with a gun did what we wanted), it would appear to be the right solution.