late-dentist-52023
05/19/2023, 9:02 PMThe number of conversions required in an experiment variation before showing results (default 150)
For a duration metric the minimum sample size is described as:
The total duration required in an experiment variation before showing results (default 02:30)
So binomial appears to apply as a count, but duration is dependent on the value. If we are testing small values (e.g. load times, things in milliseconds) and were to use avg(time_in_ms/1000.0)
as our user value aggregation in an attempt to understand if an experiment has a positive or negative impact on load time — how does that relate to the total duration for the minimum sample size — should we just set that setting to 0
? e.g. if our average user saw 0.1 seconds, do I need 1500 average users to have reached that total duration threshold or is there a different interpretation?helpful-application-7107
05/19/2023, 9:37 PMsum
of user-level values that is less the min sample size. If the sum
of the user-level aggregates for either variation is > than min sample size, then we display the results in the front-end.if our average user saw 0.1 seconds, do I need 1500 average users to have reached that total duration thresholdYes, but this would be per variation, roughly.
late-dentist-52023
05/19/2023, 9:43 PMhelpful-application-7107
05/19/2023, 9:45 PMlate-dentist-52023
05/19/2023, 9:45 PMhelpful-application-7107
05/19/2023, 9:45 PMDoes that also means that at least 1 user in each variant with a very large value (real or error) then (e.g. 1 user with 3600 seconds of value) would also be sufficient for results to display?That's correct. You could use hard capping (e.g. winsorization) to prevent this, and we're working on adding percentile capping.
late-dentist-52023
05/19/2023, 9:46 PMhelpful-application-7107
05/19/2023, 9:47 PMlate-dentist-52023
05/19/2023, 9:49 PMhelpful-application-7107
05/19/2023, 9:50 PM