It's About Time: What A/B Test Metrics Estimate
It's About Time: What A/B Test Metrics Estimate
Online controlled experiments, or A/B tests, are large-scale randomized trials in digital environments. This paper investigates the estimands of the difference-in-means estimator in these experiments, focusing on scenarios with repeated measurements on users. We compare cumulative metrics that use all post-exposure data for each user to windowed metrics that measure …