Any thoughts on how you would run cluster randomized experiments in GrowthBook?

Hey Xavier! You can run cluster randomized experiments right now, but we don't have a good way to analyze data at the sub-cluster level (you'd have to adjust your standard errors in the Frequentist engine, or model your clusters in the bayesian engine, which we don't have support for right now). Some options... Option 1 (randomize and analyze at the cluster level): 1. Randomize and experiment: Create a cluster ID and randomize using that cluster ID (e.g. set up a feature on that cluster ID). 2. Measure Metrics a. Option a: define metrics at the cluster level, and write custom SQL to ensure they map to the metrics you care about. b. Option b: keep your metrics at the user level (assuming you already have them) and create a user -> cluster mapping in your Datasource so that we can re-aggregate at the cluster level automatically in your experiment query This will provide statistically valid estimates at the cluster level, but the main drawbacks are: (1) your power will be likely be severely limited if you have few clusters and (2) you may care about effects at the user level instead of at the cluster level Option 2 (hackier, and with incorrect statistical properties in experiment analysis): 1. Randomize and experiment: Randomize with the

hashAttribute

set to the cluster ID rather than the default

id

. with some cluster ID so that randomization is consistent within cluster, but also track the user ID as an attribute with the trackingCallback 2. Then just analyze metrics at the user level (to be honest I am not 100% certain this will work, but I believe it should as the tracking callbacks from step one should still have the user id as well). Of course, the main issue here is that your uncertainty will likely be greatly underestimated and the statistics we produce will be largely invalid. Ideally between the two (in terms of statistical power) would be official support for cluster randomization and while on our radar, is not high priority right now.

Thanks for the quick answer! We are interested in the Bayesian approach so any preference on option 1 or 2? Initially I was thinking about recommending option 2 but after your explanation I think option 1 might be better?

So long as you track both user and cluster id, then you could try both approaches simultaneously on the analysis side, and I'd be happy to help out with implementation as I'm curious about it myself.

Cool! Thanks for the offering. We might take your offer to help out with implementation. The person interested in running this type of experiment is out for the rest of the week so we'll get back to you next week at the earliest.

Sure, just ping me here!