This message was deleted.

Whatever you can measure and associate with the variation you can compare. Page load time is a number. The first difficult part probably is how to implement technically collecting that information (page load time in the scope of the experiment). The second moment is the calculation - if your splitting unit is user and analytical unit is page, then if you have multiple pageviews by user you are dealing with ratio metric (total_time/pageviews). You'll need to keep it in mind when creating the metric. Just my two cents (I'm not from the GB team).

Hi Yury. Yes, all your mentioned potential issues have been addressed. I am just not sure how to set up test correctly so that I get the desired false negative and false positive rate. When monitoring one single metric without guardrails - the calculations are straightforward. When guardrails get into play - that’s where I get confused. I checked this articles that describes guardrails framework in AirBnb but there’s nothing about error rate control https://medium.com/airbnb-engineering/designing-experimentation-guardrails-ed6a976ec669 Though they do mention that multiple guardrail metrics lead to higher probability of “false alarm”

Guardrail metric is just another metric, nothing special, you are just making sure it's not moving into unwanted direction. You can apply the same calculation for Type 1 and 2 errors as to any other metric when doing power analysis. If you don't want to dive into all the complexity related to ratio metrics, you can just compare two data sets after calculating averages for each user. In some cases it's not the right thing to do, but in your case I think it would be alright, as you are probably interested in avoiding rather big move of the guardrail metric. So it's just as simple/complicated as getting power analysis for two numeric datasets (variations) of page load times (average per user) using any statsig tool.