Hey, have a issue with a url redirect experiment. ...
# experimentationn
nutritious-market-77567
08/12/2024, 8:01 AMHey, have a issue with a url redirect experiment. SRM is really high and consistent over time.
What would be the steps to troubleshoot this? I'd rather relaunch the experiment.
Attaching some screenshots.
f
flaky-noon-11399
08/14/2024, 9:03 AMHi Radvile, good morning! Apologies for the delayed response and thank you for your patience. Please can you confirm which SDK you are using to build this redirect test?
flaky-noon-11399
08/14/2024, 9:03 AMAlso are you using GTM in your implementation?
flaky-noon-11399
08/14/2024, 9:04 AMIf so, what you've described is a known problem with URL Redirect tests, and the issue is especially pronounced when using Google Tag Manager (GTM). Not sure if you're using that or not.
flaky-noon-11399
08/14/2024, 9:04 AMWhat happens is that the inherent latency of the trackingCallback causes the URL Redirect to take place before the trackingCallback can complete, which means a much larger exposure to the control URL than to the redirect URL.
flaky-noon-11399
08/14/2024, 9:05 AMWe have a PR open to help address this for some common use cases, and this PR will be particularly useful for folks using the HTML Script Tag SDK. It won't completely eliminate the problem, but will make it less egregious.
flaky-noon-11399
08/14/2024, 9:06 AMIn the interim, you can try increasing the amount of time the trackingCallback has in order to complete by setting the
context.navigateDelayflaky-noon-11399
08/14/2024, 9:07 AMHowever, the PR is scheduled to be merged late this week or early next week.
flaky-noon-11399
08/14/2024, 9:08 AMIf performance is critical, we recommend doing A/B testing using Edge Workers instead of on the client. We have an example using Cloudflare here, but also support Fastly and AWS.
flaky-noon-11399
08/14/2024, 9:08 AMAs for the PR, for some tracking libraries like GA4 and GTM, GrowthBook waits until the tracking call finishes. So this will likely be much faster than 300ms unless the user is on a very slow network. For other tracking libraries like Segment, we follow their best practices and wait a standard 300ms.
flaky-noon-11399
08/14/2024, 9:08 AMIt's also worth mentioning that this problem with URL Redirects, especially when used in combination with GTM, is not unique to GrowthBook. Any A/B testing tool that supports client-side redirects will have this same problem.
flaky-noon-11399
08/14/2024, 9:09 AMHope this helps and sorry for the lengthy response, we wanted to make sure you had all context 🙏
n
nutritious-market-77567
08/14/2024, 12:44 PMHey @flaky-noon-11399 thanks for all this.
We use a mix of direct implementation (HTML Script Tag SDK) + GTM implementation (in-app, where most of conversion happen). I'll talk to our developers and share what you shared with me and see what's the next best step forward. I'll come back here if I have any follow up questions.
f
flaky-noon-11399
08/14/2024, 12:45 PMHi @nutritious-market-77567 it was my absolute pleasure to assist and yes this would definitely be the culprit given your implementation of GTM and the HTML SDK.
43 Views