This message was deleted.
# give-feedbackw
worried-planet-2191
05/19/2023, 9:02 PMThis message was deleted.
h
helpful-application-7107
05/19/2023, 9:37 PMMinimum sample size largely functions to prevent the experiment results from displaying results prematurely. If your metric values are very small (as it appears they will be for your average load time), then you probably do want to lower your minimum sample size.
To your initial question, the minimum sample size will prevent the front-end showing results when BOTH comparison variations (e.g. control and treatment) have a
sumsumhelpful-application-7107
05/19/2023, 9:37 PMif our average user saw 0.1 seconds, do I need 1500 average users to have reached that total duration thresholdYes, but this would be per variation, roughly.
helpful-application-7107
05/19/2023, 9:39 PMYou can set it to 0 and then just rely on the people interpreting results to have the patience to wait for enough users to be bucketed, rather than set a min sample size, or you can set it to some larger value. If you set the min sample size, to, say 10, and your average is 0.1s, then you should start to see results when either variation gets around ~100 users, depending on the actual values you get (as per your original interpretation)
l
late-dentist-52023
05/19/2023, 9:43 PM👍 thanks for confirming, I was expecting “sample size” to refer to the number of users (or potential exposures in a multiple exposures context) included rather than the sum of the duration. e.g. if 150 users in each cohort were averaging 0.1s I would have expected that to have met the threshold the same as 150 users averaging 1000 seconds. Does that also means that at least 1 user in each variant with a very large value (real or error) then (e.g. 1 user with 3600 seconds of value) would also be sufficient for results to display?
h
helpful-application-7107
05/19/2023, 9:45 PMThe name is (and I'm guessing here) likely a hold over from when it was targeted at binomial metrics, but I agree it seems like it should refer to users and we could improve that.
l
late-dentist-52023
05/19/2023, 9:45 PMWe have both scenarios, things like that “load time” concept as well as video consumption which can be hours worth of time.
h
helpful-application-7107
05/19/2023, 9:45 PMDoes that also means that at least 1 user in each variant with a very large value (real or error) then (e.g. 1 user with 3600 seconds of value) would also be sufficient for results to display?That's correct. You could use hard capping (e.g. winsorization) to prevent this, and we're working on adding percentile capping.
l
late-dentist-52023
05/19/2023, 9:46 PMđź‘Ť ok, that helps, we can use that to adjust the configuration for our duration metrics.
h
helpful-application-7107
05/19/2023, 9:47 PMGreat! Sorry for the lack of clarity but hopefully the levers are there to help you get the functionality you want. I'll note down the UX concerns and if there's some functionality that would be nice to have, definitely let us know.
l
late-dentist-52023
05/19/2023, 9:49 PMSure, having the values not share the same global default when a metric is created (but one translates the default of “150” into 2 minutes 30 seconds) could be one approach.
h
helpful-application-7107
05/19/2023, 9:50 PMAgreed.
139 Views