Hi, I’m running an A/A test to validate my impleme...
# ask-questions
s
Hi, I’m running an A/A test to validate my implementation using Cloudflare Workers at the edge. After about 5–7 hours, the disconnection rate starts increasing until the page becomes unavailable. A rate below 3% is acceptable, but it’s climbing to around 25%. At the same time, I’ve noticed that the Worker wall time keeps increasing. Has anyone run into this issue before? I’m out of ideas on how to fix it.
f
I'll ping the developer who worked on that
s
Thanks 🙏
f
1. What sort of payload caching are you doing? KV store with SDK webhooks, or letting the worker fetch defs into mem. The latter is super inefficient. 2. What kind of tests are you doing? If only redirects, you might configure
RUN_VISUAL_EDITOR_EXPERIMENTS="browser"
or
"skip"
, and set
ALWAYS_PARSE_DOM=false
. Could also try internalizing tracking callbacks and setting
DISABLE_INJECTIONS=true
but that will require more setup to ensure user id assignment is sticky.
s
1. KV Payload Cache (KV_GB_CACHE). 2. I’m using a feature flag. Since it’s an A/A test with no redirects, I went with this approach.
If I redeploy the worker, the problem is gone for the next 5-7 hours.
I think the it could be accumulated “fetch” connections locked due to a lack of timeout, but fixing this not worked.
m
Hey Gabriel. I'm a Solution Engineer here at GrowthBook. Maybe I can help you debug. What kind of volume are you sending the worker to see these delays? What do you mean by "disconnection rate"?
s
Hi, the Worker intercepts requests to the website, handling around 1,000 users per hour. When I say "disconnection rate", I'm referring to the
clientDisconnected
status in Cloudflare Workers. It's as if users try to access the page, but it keeps loading until it times out. This starts happening about 5–7 hours after the Worker is deployed.
m
Can you share your edge worker source? Are you running your own GB instance as the origin or using GB Cloud?
s
Cloud. I can't share it because it's in a corporate monorepo. Today I changed the caching strategy to use webhooks, so I'll test it again tomorrow. If it still doesn't work, I'll create an equivalent repository so you can take a look, ok?
👍 1
m
Sounds good. Or maybe you could just zip it up and share it. Shouldn't be many files. I'm assuming you're using the cloudflare implementation from GrowthBook (you could be building your own on our base layer).
Any luck? FYI, I'll be OOO tomorrow through Monday, back on Tuesday.
s
So far, it seems to have worked. I’m monitoring it.
m
Nice! 🙂 If it continues to look good, I'd love to see some details as to what you changed so we can figure out on our side what the cause was.
s
Ok.
I think that if it keeps working properly through the end of the day, I’ll consider the issue resolved. Basically, what I did was change the cache strategy to use webhooks, so there are far fewer write operations.
m
Webhooks to update the cached features from the GB servers?
s
From cloudflare kv
I changed from “kv payload cache” to “kv stored payloads”.
m
Interesting. You think your traffic just led to enough writes to slow down the worker? That would have to be a lot of writes. The webhook approach is less writes but still seems strange this fixed your issue. Would be nice to know what the slow down was actually caused by.
s
Hi Chris. The worker behavior is still stable, so I will consider the issue solved. It doesn't make sense to me, but worked here hahaha.
Traffic was heavy, but it shouldn't have caused this behavior. The TTL was set to 15 min. One thing I did was customize some edge plugin fetches to add a timeout to them and force-close the request body. None of this solved the issue, but I left it in the code.