Choice of the Randomization Unit in Online Controlled Experiment
Deng, Longbotham, Walker, Xu
Summary
This paper discusses pros and cons of choosing User / Page view as Randomization unit / Analysis unit. Choosing user as randomization unit can deliver consistent experience for users. Choosing page view as randomization unit reduces the variance and achieves better power for analysis, but it loses availability of many user level metrics, such as sessions per unique user, page loading time per unique users, etc.
This paper derives the asymptotic variance for different scenarios with User / Page view as Randomization unit / Analysis unit.
Key Concept:
Randomization Unit: In a randomized experiment, we call the unit on which randomization is performed the randomization unit.
Analysis Unit: In analysis phase, a metric will be naturally associated with an unit, which we call it analysis unit.
User and Page view are mostly used Randomization Unit and Analysis Unit.
Variance Analysis:
Suppose X_ij is the per-page measurement on user i's jth pageview.
i = 1,...,n; j=1,...K_i
X_ij ~ (u_i, sigma^2_i)
Consider the t-test statistic:
(mean(X_T) - mean(X_C)) / sqrt(Var(mean(X_T) - mean(X_C)))
Randomization Unit: User + Analysis Unit: User
Simulation model: Draw (u_i, sigma^2_i) for each user, and draw X_i ~ (u_i, sigma^2_i).
mean(X_T) = mean(X_i for i in Treatment group)
The test is Regular t-test
mean(X_T) and mean(X_C) are independent
Var(mean(X_T) - mean(X_C)) = Var(mean(X_T)) - Var(mean(X_C))
Var(mean(X_T)), Var(mean(X_C)) can be estimated by sample variance.
Randomization Unit: User + Analysis Unit: Pageview
Simulation model: Draw (u_i, sigma^2_i) for each user, and draw X_ij ~ (u_i, sigma^2_i) i.i.d.
mean(X_T) = mean(X_ij for i in Treatment group, j=1,...,K_i)
Note that X_ij are independent conditioned on (u_i, sigma^2_i).
mean(X_T) and mean(X_C) are independent.
Var(mean(X_T)), Var(mean(X_C)) can be estimated by direct method (asymptotically biased) or Delta method (asymptotically unbiased).
Theorem 1
n * Var (mean(X)) -> C Var(u_i) + E(sigma^2_i) / E(K_i)
where C = E(K_i^2)/(EK_i)^2.
C Var(u_i): user effect, between user variance.
E(sigma^2_i) / E(K_i): Variance not explained by user effect, within user variance.
When EK_i large, between user variance will dominate.
Randomization Unit: Pageview + Analysis Unit: User
Not applicable. All page views are randomly divided into different groups. Since the data for the same user will be split into both treatment group and control group. It is not designed to make inference on user level.
Randomization Unit: Pageview + Analysis Unit: Pageview
Simulation model 1 (no user effect):
All page views from all possible users are randomly divided into different groups. Treat page view level measurement as i.i.d., since the page views are drawn from all users and no user selection variance is induced in this randomization scheme.
Given the i.i.d. page view level measurements, analysis is straightforward. - Regular t-test.
Simulation model 2 (user effect as random effect):
Two Layer Randomization Framework
First draw n users (like random effect); Then generate page view level measurements and divided into different groups.
User variance will show up for both within-group variance and between-group variance.
mean(X_T) = mean(X_ij for (i,j) in Treatment group)
mean(X_T) and mean(X_C) are NOT independent, since they come from common users.
Within each group, the variance can be estimated by the same way (taking the same form) as User + Pageview by direct method or Delta method.
Theorem 5
n * Var (mean(X_1) - mean(X_2)) -> (1/EK_i(1) + 1/EK_i(2)) (Var(u_i) + E(sigma^2_i))
Asymptotically unbiased estimator: direct_estimator(Var (mean(X_1))) + direct_estimator(Var (mean(X_2)))
Interesting observation:
For variance estimator,
Under Randomization Unit: User + Analysis Unit: Pageview,
Delta is asymptotically unbiased
Direct method is asymptotically biased
Under Randomization Unit: Pageview + Analysis Unit: Pageview, with two layer model
Delta is asymptotically biased
Direct method is asymptotically unbiased
Deng, Longbotham, Walker, Xu
Summary
This paper discusses pros and cons of choosing User / Page view as Randomization unit / Analysis unit. Choosing user as randomization unit can deliver consistent experience for users. Choosing page view as randomization unit reduces the variance and achieves better power for analysis, but it loses availability of many user level metrics, such as sessions per unique user, page loading time per unique users, etc.
This paper derives the asymptotic variance for different scenarios with User / Page view as Randomization unit / Analysis unit.
Key Concept:
Randomization Unit: In a randomized experiment, we call the unit on which randomization is performed the randomization unit.
Analysis Unit: In analysis phase, a metric will be naturally associated with an unit, which we call it analysis unit.
User and Page view are mostly used Randomization Unit and Analysis Unit.
Variance Analysis:
Suppose X_ij is the per-page measurement on user i's jth pageview.
i = 1,...,n; j=1,...K_i
X_ij ~ (u_i, sigma^2_i)
Consider the t-test statistic:
(mean(X_T) - mean(X_C)) / sqrt(Var(mean(X_T) - mean(X_C)))
Randomization Unit: User + Analysis Unit: User
Simulation model: Draw (u_i, sigma^2_i) for each user, and draw X_i ~ (u_i, sigma^2_i).
mean(X_T) = mean(X_i for i in Treatment group)
The test is Regular t-test
mean(X_T) and mean(X_C) are independent
Var(mean(X_T) - mean(X_C)) = Var(mean(X_T)) - Var(mean(X_C))
Var(mean(X_T)), Var(mean(X_C)) can be estimated by sample variance.
Randomization Unit: User + Analysis Unit: Pageview
Simulation model: Draw (u_i, sigma^2_i) for each user, and draw X_ij ~ (u_i, sigma^2_i) i.i.d.
mean(X_T) = mean(X_ij for i in Treatment group, j=1,...,K_i)
Note that X_ij are independent conditioned on (u_i, sigma^2_i).
mean(X_T) and mean(X_C) are independent.
Var(mean(X_T)), Var(mean(X_C)) can be estimated by direct method (asymptotically biased) or Delta method (asymptotically unbiased).
Theorem 1
n * Var (mean(X)) -> C Var(u_i) + E(sigma^2_i) / E(K_i)
where C = E(K_i^2)/(EK_i)^2.
C Var(u_i): user effect, between user variance.
E(sigma^2_i) / E(K_i): Variance not explained by user effect, within user variance.
When EK_i large, between user variance will dominate.
Randomization Unit: Pageview + Analysis Unit: User
Not applicable. All page views are randomly divided into different groups. Since the data for the same user will be split into both treatment group and control group. It is not designed to make inference on user level.
Randomization Unit: Pageview + Analysis Unit: Pageview
Simulation model 1 (no user effect):
All page views from all possible users are randomly divided into different groups. Treat page view level measurement as i.i.d., since the page views are drawn from all users and no user selection variance is induced in this randomization scheme.
Given the i.i.d. page view level measurements, analysis is straightforward. - Regular t-test.
Simulation model 2 (user effect as random effect):
Two Layer Randomization Framework
First draw n users (like random effect); Then generate page view level measurements and divided into different groups.
User variance will show up for both within-group variance and between-group variance.
mean(X_T) = mean(X_ij for (i,j) in Treatment group)
mean(X_T) and mean(X_C) are NOT independent, since they come from common users.
Within each group, the variance can be estimated by the same way (taking the same form) as User + Pageview by direct method or Delta method.
Theorem 5
n * Var (mean(X_1) - mean(X_2)) -> (1/EK_i(1) + 1/EK_i(2)) (Var(u_i) + E(sigma^2_i))
Asymptotically unbiased estimator: direct_estimator(Var (mean(X_1))) + direct_estimator(Var (mean(X_2)))
Interesting observation:
For variance estimator,
Under Randomization Unit: User + Analysis Unit: Pageview,
Delta is asymptotically unbiased
Direct method is asymptotically biased
Under Randomization Unit: Pageview + Analysis Unit: Pageview, with two layer model
Delta is asymptotically biased
Direct method is asymptotically unbiased
No comments:
Post a Comment