Experiments at Airbnb
http://nerds.airbnb.com/experiments-at-airbnb/
How long should experiments run for?
1, Power calculation. Here is a resource that helps with that computation.
2, See the trend of delta or p-value overtime, and determine whether it converts, rather than to consider the single result of an effect with a p-value. Then make conclusion.
3, When the results are extremely good/bad, may terminate early.
Be skeptical of early results
4, Dynamic p-value threshold.
"We solved the problem of how to figure out the p-value threshold at which to stop an experiment by running simulations and deriving a curve that gives us a dynamic (in time) p-value threshold to determine whether or not an early result is worth investigating. We wrote code to simulate our ecosystem with various parameters and used this to run many simulations with varying values for parameters like the real effect size, variance and different levels of certainty. This gives us an indication of how likely it is to see false positives or false negatives, and also how far off the estimated effect size is in case of a true positive. In Figure 6 we show an example decision boundary."
http://nerds.airbnb.com/experiments-at-airbnb/
How long should experiments run for?
1, Power calculation. Here is a resource that helps with that computation.
2, See the trend of delta or p-value overtime, and determine whether it converts, rather than to consider the single result of an effect with a p-value. Then make conclusion.
3, When the results are extremely good/bad, may terminate early.
Be skeptical of early results
4, Dynamic p-value threshold.
"We solved the problem of how to figure out the p-value threshold at which to stop an experiment by running simulations and deriving a curve that gives us a dynamic (in time) p-value threshold to determine whether or not an early result is worth investigating. We wrote code to simulate our ecosystem with various parameters and used this to run many simulations with varying values for parameters like the real effect size, variance and different levels of certainty. This gives us an indication of how likely it is to see false positives or false negatives, and also how far off the estimated effect size is in case of a true positive. In Figure 6 we show an example decision boundary."
No comments:
Post a Comment