alt Hacker NewsA lot of the time you are running experiments and seeing how the data changes, not looking at the data in a vacuum.
Many of the example critiques here don't apply so much when looking at the changes in data:
- if I got 50% more hits on my site this week vs last week, that's meaningful despite 36% of people blocking ads
- if my open rate doubled when I changed my email subject, also meaningful
The other examples are hard to pick holes in as they simply say "Z is a lie", but I can be looking at multiple data sources to decide how much of a lie Z is.
Replies
N_Lens • today at 9:11 PM
Following this kind of process blindly and optimising for it leads to a terrible product. However it can lead to more revenue in the short term.
➕ show 1 reply
Having been at smaller companies without the data, tooling, discipline, and resourcing to conduct viable experiments, and then being at a company that is actually one of the best in the world at it and building solutions for these problems at scale for marketing teams, so many problems come back to the human element in how data is tracked, how teams collaborate or fail to which can lead to pollution and dirty data, and how decisions are made as the moment trade-offs need to be considered it becomes personal and enters messy human relationship territory.
But ultimately from a pure "did this work or not" standpoint you are right. Incrementality experiments are the gold standard.