https://www.growthbook.io/
Join Slack
Hey everyone, we have been running a <https://growthbook.groover.co/experiment/exp_19g613lq5a306k|te...
# ask-questions
c
Hey everyone, we have been running a test with an 80% 
control
/ 10% treatment cell A (
stars
) / 10% treatment cell B (
satisfaction-rate
) split. The 
satisfaction-rate
cell was performing somewhat poorly, so we removed users from the cell and switched the test to be 70% control, 30% treatment cell A (
stars
). After this change, I’m now seeing the 
satisfaction-rate
metrics plummeting, and many users removed due to multiple exposure errors. What I would expect is for 
satisfaction-rate
metrics to remain unchanged for the rest of the test, and for 
stars
users not to be impacted. I’m concerned that the results won’t be valid given what I’m seeing in Growthbook. Do you think anything is messed up here / Is there anything I should change from my end to reflect what we’re expecting?
f
reassigning the splits will cause a lot of multiple exposures
what you might have is a few noisy users, or some bad experiences which have now shifted to another variant
you could try seeing if playing with winsorization / capping values
c
A couple questions: 1. What is winsorization / capping values? 2. If I create a new phase, is it possible to have that phase start a week ago instead of today? Thanks @fresh-football-47124!
r
Hi Erica, winsorization, also known as capping, is a statistical technique used to limit extreme values in the data to reduce the effect of outliers. It is applied to the data by ensuring that all aggregate unit (e.g., user) values are no more than some specified value. For example, if you have a dataset where the normal revenue per user is $40, but there is an outlier with a $5000 order, winsorization can cap this value at a predetermined level, such as $100, to minimize its impact on the experiment results.
In the context of GrowthBook, winsorization is used to prevent large outliers from having an outsized effect on experiment results. This is particularly useful when you have metrics like revenue per user, where a single large purchase can skew the results. By capping the aggregated value for each user at a certain threshold, you can still count the outlier's contribution without letting it dominate the outcome of the experiment.
Yes, it is possible to manually add a new experiment phase with a start date in the past. If you don't have the ability to choose a date in the past when you create the new phase, go ahead and let it default to today, then once the new phase appears you can click the "Edit" button and modify the start date. (This is what happened to me just now when I tried it on a demo app.) Keep in mind that when you set a new start date for a phase, it is primarily for analysis purposes and does not retroactively affect the traffic that was already exposed to the experiment.
c
Thanks so much!!
73 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by