Hi everyone , if i'm already running an a/b test e...
# announcementsd
dry-eye-65136
05/10/2023, 4:06 PMHi everyone , if i'm already running an a/b test experiment and i decided to start using namespaces with the same experiment , would it cause users to switch variants ?
f
fresh-football-47124
05/10/2023, 6:05 PM
no, they won’t switch, but some people who were included may be excluded
d
dry-eye-65136
05/11/2023, 9:10 AMyes exactly that's what i'm asking about for people who were already included
dry-eye-65136
05/11/2023, 9:11 AMso may be the challenge here , at least from what i've seen is that the decision to either use namespaces or not is always based on the next experiment, which is not necessarily be known at this time
dry-eye-65136
05/11/2023, 9:12 AMjust asking if any sort of practice to do so i'm able may be to make this decision later on?
dry-eye-65136
05/11/2023, 9:13 AMmay be would i use the namespaces from day 1 , set the experiment exposure to 100% , and use the namespace splitting as the experiment exposure , not sure if this would work @fresh-football-47124 ?
dry-eye-65136
05/13/2023, 5:43 PM@fresh-football-47124 not sure if the above practice could work
f
fresh-football-47124
05/13/2023, 5:44 PM
you’re doing 100% of the namespace or exp exposure?
d
dry-eye-65136
05/13/2023, 5:46 PMNo I mean to solve the problem of deciding if I want to include other experiments in the same namespace later ( if I don't know now if I want to ) , to always use namespace anyways and use it as the experiment exposure instead
dry-eye-65136
05/13/2023, 5:46 PMThe problem is that when running an experiment, its hard to decide if I want to run another experiment within the same namespace now
dry-eye-65136
05/13/2023, 5:49 PMI only know when I plan for the next experiment to run , and discuss with the rest of stakeholders either to run it exclusively or not
f
fresh-football-47124
05/13/2023, 5:49 PM
right
fresh-football-47124
05/13/2023, 5:49 PM
we find that most people over estimate how much experiments will interact
fresh-football-47124
05/13/2023, 5:50 PM
meaningful interaction effects are usually rare
d
dry-eye-65136
05/13/2023, 5:54 PMThis is interesting!
dry-eye-65136
05/13/2023, 5:55 PMPerfect, so the above approach would work even if I'm not adding other experiments later
f
fresh-football-47124
05/13/2023, 5:56 PM
how much of the namespace are you using for each experiment?
d
dry-eye-65136
05/13/2023, 5:59 PMCurrently I have one experiment without using namespaces at all , I'm thinking of an approach be able to decide later while pushing more experiments to be able to decide either to run it exclusively or not
dry-eye-65136
05/13/2023, 6:01 PMThe question is if for example the current experiment exposure is 20% without namespaces , if I changed to use a namespace with the same key and allocated 20% of the namespace to that experiment, then set the experiment exposure to 100% , would this still cause issues making people switch variants ?
f
fresh-football-47124
05/13/2023, 6:02 PM
ah, yes, I think it would - I think we use a different hashing seed for the namespace on purpose
fresh-football-47124
05/13/2023, 6:03 PM
But let me check the code
👍 1
fresh-football-47124
05/13/2023, 6:06 PM
ya, when hashing namespaces we change the seed
d
dry-eye-65136
05/14/2023, 5:32 AMGreat , this is good to know thanks a lot
30 Views