[Reading Notes] Overlapping Experiment Infrastructure

Overlapping Experiment Infrastructure: More, Better, Faster Experimentation
KDD 2010
[Google] Tang, Agarwal, O'Brien, Meyer

Conventional experiment infrastructure:
Single layer of experiments: every query is in at most one experiment.
Pros: easy to use, flexible
Cons: insufficiently scalable
Multi-factorial experiment design: each parameter can be experimented on independently.
However, some parameters can not be independent, especially when the number of parameters is large. E.g. blue text cannot be combined with blue background.

Overlapping experiment infrastructure
Advantages: easy to use, speed, scalability, flexibility, robustness, enable gradual ramping-up of launches
Parameter partition: 
parameters from different binaries can be in different subsets, which solves the starvation and bias issues.
all parameters for a given binary do not need to be in a single subset, that can be further partitioned.


Key concepts:
domain: a segmentation of traffic
layer: corresponds to a subset of the system parameters.
experiment: a segmentation of traffic where zero or more system parameters can be given algternate values
Relations: domain contain layers, layers contain experiments and can also contain domains. domains and layers can be nested.
Launch layers are always contained within the default domain (i.e. they run over all traffic).
A parameter can be in at most one launch layer and at most one normal layer (within a domain) simultaneously.
Alternative default values of parameters can be specified in launch layer.
Order of adopting parameter values (if specified in higher order, lower order values will be ignored):
Normal experiment layer parameter > Launch layer default value > system default value
Usage of the Launch layer: 1, gradually roll out changes to all users and to delete that layer when the feature is fully rolled out. 2, experiments in launch layers are generally larger, they can be used to test for interactions between features.

Traffic diversion types:
cookie-mods
random traffic
user-id mods
cookie-day mods: combine the mod of the cookie with the day, the set of cookies in an experiment changes from day to day.
Divert order: user id, cookie, cookie-day, random traffic

After diverting traffic, conditions provide better utilization of this traffic by only assigning specific events to an experiment or domain.
Google supports conditions based on country language, browser, etc.
Usage: 1, for particular population, e.g. Japan users; 2, canary new code, test new code on a small amount of traffic.

Biased / Unbiased traffic
If we take all traffic corresponding to a particular cookie mod, it is unbiased.
If after condition, take the rest of the traffic for subsequent diversion types, it is biased traffic. This should be avoided. Tagging the unassigned traffic with a biased id.

Other topics covered by this paper:
Procedure of evaluating and launching a typical feature (end of section 4)
sample size 5.2.1
triggering, logging, counter-factuals 5.2.2
pre-, post-periods (A/A test) 5.2.3


"Each binary has an associated binary push and data push."
Question: What is binary? Is that a program block? The output seems not a binary one.