Another question; we currently have 1M+ MAU (F2P game). Something I'm trying to work out is whether the math changes for A/B tests when only 1 in 10 users are in the sample set, or if we can just run tests as normal. In other words, say only 1:10 users are recorded in the data warehouse to limit data warehousing costs and we run A/B tests on 100% of those users. Do we have additional error bars on our test results to account for population sample error?

Are you exposing all users to the experiment on the front end and then only tracking 10% of them in the data warehouse?

With our setup we could either do full exposure or exposure to just the tracked cohort. (Right now we're running tests on the full population using Game Analytics, which is extremely limited but free)

Ok, I'd probably recommend only exposing the tracked cohort to an experiment. As long as your sampling is truly random, it shouldn't be an issue statistics wise.