Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained.
KDD 2012
Kohavi, Deng, Frasca, Longbotham, Walker, Xu
Rate of Success
1/3: One third of ideas tested at Microsoft improved the metric(s) they were designed to improve.
10%: Google ran approximately 12,000 randomized experiments in 2009, with only about 10 percent of these leading to business changes.
10%: Netflix considers 90% of what they try to be wrong.
33% guess to be true: Regis Hadiaris: I've been doing this for 5 years, and I can only "guess" the outcome of a test about 33% of the time.
Puzzling case 1 The OEC (Overall Evaluation Criterion) for Search Engine
Puzzling outcome: Poor searching results lead to two metrics improved significantly: Distinct queries per user, Revenue per user.
Why? Inappropriate OEC.
Explaination:
Revenue per user: When poor search results show, the relative relevance of ads increases, so users click more on ads. Revenue per user should not be used as an OEC for search and ad experiments without other constraints. We want this OEC increase without negatively impacting engagements metrics like sessions/user.
Distinct queries per user = Distinct queries per session * Sessions per user: Distinct queries per task should be minimized, but it is hard to measure. Distinct queries per session is subtle. Increasing it may indicate more queries are needed to complete a task, and decreasing it may indicate abandonment. This should be minimized subject to the task being successfully completed. Sessions per user is the key metric to optimize the experiments, as satisfied users will come more. (Tasks per user is preferrable but it is hard to measure.)
Solution:
Using Sessions per user as OEC.
In addition, we can do: minimizing distinct queries per user conditional on task completed, maximizing revenue per user conditional on no negative impact on long-term user value.
Lesson: Short term metrics can be inconsistent with long term goal.
Puzzling case 2 Click Tracking
Puzzling outcome: An additional code was added that slowed down the user experience slightly, but the experiment showed that users were clicking more.
Why? No real difference, but rather an instrumentation difference. Click-beacons have more chance to be sending back for additional time.
Explanation: Most sites use web beacons to track user actions, but returning the clicks and submits slows the next action. Chrome, Firefox, and Safari are aggressive about terminating requests, when navigating away from the current page. A non-negligible percentage of click-beacons never make it to the server. Adding a small delay gives the beacon more time, and more click request beacons reach the server. That is why the experiment showed users were clicking more.
Solution: IE continues to execute image/beacon requests even after navigation, that makes click tracking more reliable.
Lesson: Browsers can cause difference due to instrumentation issues or differences in HTML/JavaScript parsing.
Puzzling case 3 Initial Effects Appear to Trend
Puzzling outcome: The first few days' effect seems like a trend, like the figure above. Can we expect the next outcome to be positive? NO. (It is actually an A/A test.)
Why? This is not a trend, but rather caused by shrinking of the variance (and confidence interval), and cumulative effect.
Explanation: The effects in the initial days usually seem overly positive or negative, due to relatively small sample size, even if there is no real difference. During the first few days, the cumulative results seem to trend. It is rare that the Primacy effects reverse the initial effects (Effect is initially negative, when the user gets used to it, becomes positive).
Solution: High variance means that we need to collect enough data to get better estimates; early results are often misleading.
Lesson: Initial effects don't mean trend. Collect enough data to make inference.
Puzzling case 4 Experiment Length and Statistical Power
Puzzling outcome: For some of the key metrics, including Sessions/user (or the percent change of this metric), running the experiment longer does not provide additional power.
Why? Due to change of s.d. and mean over time for these metrics.
Explanation: Given the sample size and population variance are the same for treatment and control group, the width of the confidence interval for percentage change is roughly proportional to
CV/sqrt(sample size)
CV = S.D. / Mean
If CV is constant over time, then CI shrinks over time. If CV changes over time, e.g. for sessions/user (both S.D. and mean increase), CI may not shrink and power may not increase.
Solution: When looking at these metrics, we must run the experiments with more users per day. (Running experiment with more users per day is different from running longer.) Even running the experiment longer does not increase power, we should still run the experiment more than a week, for day-of-week effects, and primacy and novelty effects (through rare).
Lesson: For metrics, we need investigate whether the S.D. and mean will change over time. And if so, it means running the experiment longer will not necessarily provide additional power. Including more users per day is suggested.
Question: So, does the CV change over time for percentage change of Sessions/user?
Puzzling case 5 Carryover Effects
Puzzling outcome: For bucket experiment, some unrelated metrics may be significant, but rerunning the experiment will find these effects disappeared.
Why? These effects are carried over from the previous experiment done in the bucket.
Explanation: Carryover effect is one big drawback of the bucket system: the same users who were impacted by the first experiment are being used for the follow-on experiment.
Solution: A/A test can be used to check for carryover effects.
One way to solve the issue is by local randomization.
(1) Re-randomizing users by changing the hashing function. But we need to stop all running experiments in that bucket line to change the hashing function.
(2) Two-level bucket system. Only re-randomizing on a subset of buckets by changing the hash function seed at second level. Drawback: can't use a shared control, each experiment needs its own control. Benefit: Retrospective A/A experiment. By changing the hashing function, the experiment running previously can be evaluated as an A/A experiment for the later A/B experiment. If A/A experiment shows an effect for key metrics, then change the hashing key and retry.
Lesson: Beware of the carryover effect by bucket system. Two-level bucket system is suggested for solving the issue.
Two themes
1. Instrumentation is not as precise as we would like it to be. Click tracking.
2. Offline experiments don't always map well online. Carryover effects, confidence intervals don't shrink by extending the duration and number of users.
KDD 2012
Kohavi, Deng, Frasca, Longbotham, Walker, Xu
Rate of Success
1/3: One third of ideas tested at Microsoft improved the metric(s) they were designed to improve.
10%: Google ran approximately 12,000 randomized experiments in 2009, with only about 10 percent of these leading to business changes.
10%: Netflix considers 90% of what they try to be wrong.
33% guess to be true: Regis Hadiaris: I've been doing this for 5 years, and I can only "guess" the outcome of a test about 33% of the time.
Puzzling case 1 The OEC (Overall Evaluation Criterion) for Search Engine
Puzzling outcome: Poor searching results lead to two metrics improved significantly: Distinct queries per user, Revenue per user.
Why? Inappropriate OEC.
Explaination:
Revenue per user: When poor search results show, the relative relevance of ads increases, so users click more on ads. Revenue per user should not be used as an OEC for search and ad experiments without other constraints. We want this OEC increase without negatively impacting engagements metrics like sessions/user.
Distinct queries per user = Distinct queries per session * Sessions per user: Distinct queries per task should be minimized, but it is hard to measure. Distinct queries per session is subtle. Increasing it may indicate more queries are needed to complete a task, and decreasing it may indicate abandonment. This should be minimized subject to the task being successfully completed. Sessions per user is the key metric to optimize the experiments, as satisfied users will come more. (Tasks per user is preferrable but it is hard to measure.)
Solution:
Using Sessions per user as OEC.
In addition, we can do: minimizing distinct queries per user conditional on task completed, maximizing revenue per user conditional on no negative impact on long-term user value.
Lesson: Short term metrics can be inconsistent with long term goal.
Puzzling case 2 Click Tracking
Puzzling outcome: An additional code was added that slowed down the user experience slightly, but the experiment showed that users were clicking more.
Why? No real difference, but rather an instrumentation difference. Click-beacons have more chance to be sending back for additional time.
Explanation: Most sites use web beacons to track user actions, but returning the clicks and submits slows the next action. Chrome, Firefox, and Safari are aggressive about terminating requests, when navigating away from the current page. A non-negligible percentage of click-beacons never make it to the server. Adding a small delay gives the beacon more time, and more click request beacons reach the server. That is why the experiment showed users were clicking more.
Solution: IE continues to execute image/beacon requests even after navigation, that makes click tracking more reliable.
Lesson: Browsers can cause difference due to instrumentation issues or differences in HTML/JavaScript parsing.
Puzzling case 3 Initial Effects Appear to Trend
Puzzling outcome: The first few days' effect seems like a trend, like the figure above. Can we expect the next outcome to be positive? NO. (It is actually an A/A test.)
Why? This is not a trend, but rather caused by shrinking of the variance (and confidence interval), and cumulative effect.
Explanation: The effects in the initial days usually seem overly positive or negative, due to relatively small sample size, even if there is no real difference. During the first few days, the cumulative results seem to trend. It is rare that the Primacy effects reverse the initial effects (Effect is initially negative, when the user gets used to it, becomes positive).
Solution: High variance means that we need to collect enough data to get better estimates; early results are often misleading.
Lesson: Initial effects don't mean trend. Collect enough data to make inference.
Puzzling case 4 Experiment Length and Statistical Power
Puzzling outcome: For some of the key metrics, including Sessions/user (or the percent change of this metric), running the experiment longer does not provide additional power.
Why? Due to change of s.d. and mean over time for these metrics.
Explanation: Given the sample size and population variance are the same for treatment and control group, the width of the confidence interval for percentage change is roughly proportional to
CV/sqrt(sample size)
CV = S.D. / Mean
If CV is constant over time, then CI shrinks over time. If CV changes over time, e.g. for sessions/user (both S.D. and mean increase), CI may not shrink and power may not increase.
Solution: When looking at these metrics, we must run the experiments with more users per day. (Running experiment with more users per day is different from running longer.) Even running the experiment longer does not increase power, we should still run the experiment more than a week, for day-of-week effects, and primacy and novelty effects (through rare).
Lesson: For metrics, we need investigate whether the S.D. and mean will change over time. And if so, it means running the experiment longer will not necessarily provide additional power. Including more users per day is suggested.
Question: So, does the CV change over time for percentage change of Sessions/user?
Puzzling case 5 Carryover Effects
Puzzling outcome: For bucket experiment, some unrelated metrics may be significant, but rerunning the experiment will find these effects disappeared.
Why? These effects are carried over from the previous experiment done in the bucket.
Explanation: Carryover effect is one big drawback of the bucket system: the same users who were impacted by the first experiment are being used for the follow-on experiment.
Solution: A/A test can be used to check for carryover effects.
One way to solve the issue is by local randomization.
(1) Re-randomizing users by changing the hashing function. But we need to stop all running experiments in that bucket line to change the hashing function.
(2) Two-level bucket system. Only re-randomizing on a subset of buckets by changing the hash function seed at second level. Drawback: can't use a shared control, each experiment needs its own control. Benefit: Retrospective A/A experiment. By changing the hashing function, the experiment running previously can be evaluated as an A/A experiment for the later A/B experiment. If A/A experiment shows an effect for key metrics, then change the hashing key and retry.
Lesson: Beware of the carryover effect by bucket system. Two-level bucket system is suggested for solving the issue.
Two themes
1. Instrumentation is not as precise as we would like it to be. Click tracking.
2. Offline experiments don't always map well online. Carryover effects, confidence intervals don't shrink by extending the duration and number of users.
No comments:
Post a Comment