Hello, I have a question about a self-hosted GrowthBook instance with proxy instances using Redis as cache. All instances are running in Kubernetes. We are new to GB and during our disaster testing, we found a few problematic scenarios: Our biggest consern is that; if/when the GrowthBook API is down, the proxy instances will crash after some time and not recover until API is back up. Shouldn't the proxy instances be able to run without the API? We are using Redis as cache, so the proxy instances should be able to serve cached data even if the API is down, right?

I can check with the team on the desired behavior for the proxy server

Hey there. The proxy can be configured to stay up indefinitely while serving stale data whenever the API is down. Just set a couple of environment variables:

CACHE_ALLOW_STALE=1    # this is already on by default
CACHE_EXPIRES_TTL=999999999999    # infinity...

We'll look into refining this use case (documentation, making this the default setting).

This don't work for me. I have set the environment variables as your recomendation, but the proxy still crashes with the same error message as above when I kill the GB API. I have tested without any probes that might interfere with the proxy, but the error still occurs. Are there any other settings that I need to set to make this work?

Does the crash happen while a new instance is spinning up and the API server is already down? Or does a running instance start crashing once the API server goes down? I'll investigate further...

@happy-autumn-40938 Things look much better now, I haven't run all our test scenarios. But the proxy stays alive now 👍 Thanks for your help, and ill get in touch if we find any other issue.

great to hear it, thanks