Hi, I'm from <houseofmath.com>'s infrastructure te...
# ask-questions
s
Hi, I'm from houseofmath.com's infrastructure team, following up on an email from @fresh-football-47124 We are utilizing Growthbook Cloud, Pro and the associated Growthbook Proxy. The proxy is served via Google Cloud Platform, hosted on Cloud Run, which is basically running a proxy docker container that autoscales according to traffic and load. It is also using the Redis cache option, connected to our google managed redis instance. My setup involves a Next.js application hosted on Vercel, integrated with Growthbook through the proxy. The SDK is configured to use the proxy URL, ciphered payload (all 3 options checked), and visual experiments are enabled. The issue I'm encountering is continuous printing of the following in my logs:
Copy code
| {"level":50,"time":1698754816784,"pid":43,"hostname":"localhost","err":{"type":"SyntaxError","message":"Unexpected token < in JSON at position 0","stack":"SyntaxError: Unexpected token < in JSON at position 0\n    at JSON.parse (<anonymous>)\n    at parseJSONFromBytes (node:internal/deps/undici/undici:6662:19)\n    at successSteps (node:internal/deps/undici/undici:6636:27)\n    at node:internal/deps/undici/undici:1236:60\n    at node:internal/process/task_queues:140:7\n    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)\n    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"},"msg":"polling error"}
Additionally, on inspecting the website's network tabs, the API calls are successful through the proxy URL. However, all eventstream requests are failing, displaying
"No channel found."
Interestingly, when I use the normal Growthbook URL instead of the proxy, the eventstream works without issues. I'd appreciate any insights or guidance on this matter. I have attached some screenshots. Thank you.
βœ… 1
r
@blue-sundown-32899 has marked the issue as closed. Reopen Issue button
f
@happy-autumn-40938 any thoughts?
b
Hello, Shahadat, thanks for reaching out and for providing all he detail about what's happening. We have your support request in our queue and will reply with an update as soon as we can, likely tomorrow (Wednesday)
πŸ‘ 1
h
Let’s connect more tomorrow. Short answer though: often a proxy polling error is due to the environment variables not being set properly. I’d probably double check that they are indeed set (GROWTHBOOK_API_HOST and SECRET_API_KEY specifically)
πŸ‘€ 1
s
Thank you, Do I have the wrong API host? I am using the growthbook cloud
I am assuming this is caused by SSL. I am hosting my proxy at a domain, where ssl is auto configured by domain provider for my sdk connection, I have given proxy domain to
https
( it will always be redirected to https even with http 0) I can see env variables to let growthboob proxy handle SSL termination, but it wants a CERT and Key. As I have mentioned above, cloud run ( the service I am using ) supports streaming data How do you suggest I approach this?
h
Which version of the proxy are you using? v1.1.1 might give us a bit more debug information in the logs
Also, I believe GROWTHBOOK_API_HOST host should be
<https://api.growthbook.io>
, but not sure if this makes a real difference.
πŸ‘€ 1
s
I am using the docker image pushed 5 days ago, it doesn't seem to have versioning, only github sha
redeployed latest so that it pulls the latest release. no longer getting the long error of broken json, but getting this.
Copy code
{
  "level": 50,
  "time": 1698901060057,
  "pid": 43,
  "hostname": "localhost",
  "msg": "connection polling error: status code is 400"
}
which persists in browser network tab as well, when hitting below url
Copy code
<https://proxy.growthbook.houseofmath.com/sub/sdk-Zy6VD3oefJVPa4qZ>
h
Thanks for the update. I tested the connection polling endpoint for your account and can confirm that we're able to hit the polling endpoint (which returns 3 sdk connections). This means your environment vars are most likely set up correctly. It seems like either a networking issue or an auth issue causing your proxy to receive a 400 when polling. Can you confirm whether there is anything in the logs immediately after the "connection polling error: status code is 400" message? I don't know whether or not it's related to SSL termination. Connection polling is an outbound call from the proxy to the GB API servers. You could try setting the
NODE_TLS_REJECT_UNAUTHORIZED=0
environment variable on the proxy to see if it helps. On this point, I realized there is a bug with this parameter and have just pushed a proxy update (v1.1.3) ... it might be good to update to this version. If you did want to try terminating SSL it definitely gets trickier: You'd need to string encode the contents of your .pem files into the
HTTPS_KEY
and
HTTPS_CERT
env vars.
πŸ‘€ 1
s
I can confirm there's nothing logging immediately after 400. this is a sample section of my logs
I added the TLS reject env variable, redeploying shortly
h
Sorry, I meant the proxy container logs, not the browser.
s
this is the proxy container log. its very detailed -lists user agents as well
they are all requests of failing with
Copy code
"msg":"connection polling error
h
I don't see anything in that snippet above related to "connection polling error: status code is 400". This specific error is not initiated by a user request but rather is a recurring fetch issued from the proxy to the GB API servers... so we wouldn't expect to see a browser-like UA.
s
ah, I see. you mean these then?
using latest docker img - with
NODE_TLS_REJECT_UNAUTHORIZED=0
h
yep that's it, thanks. So the 400 returns an empty body, which smells like a networking / gateway issue
s
alright, have you seen this before for this case on google cloud, i.e how do I approach on solving this? and what problems should I expect in my site if this is not resolved? will it block my experiments? thank you for the quick responses
h
I have not seen this before unfortunately. There is definitely no requirement that you use the GrowthBook Proxy, especially if you're a cloud customer. Just set your SDK's
apiHost
to
"<https://cdn.growthbook.io>"
and everything should work just fine. As long as your SDK is able to make a successful call to the /api/features endpoint (either on cdn.growthbook.io or on your private proxy url) then all your experiments & flags will run fine.
s
Got it, and the features endpoint works, I receive a 200 response with my experiments data. This error is from the sub endpoint - failing with 400 and no channel found. Performance is a big factor in our application, thus hosting the proxy within the same region and cluster our application is being served from
h
Understood. You may want to use our CDN endpoint for now until able to get a proxy environment set up (in dev or staging preferably). The sub endpoint is for streaming (live) updates to subscribed SDKs. Streaming is a nice way to quickly push out changes to live users, although its definitely not required for basic flagging or experimentation. Our CDN endpoint also supports this streaming.
s
Thank you for the information. Do you know of any flags I can use to disable streaming temporarily?
h
sure, when instantiating the SDK, pass
backgroundSync: false
into the context.
πŸ‘Œ 1
s
got, thank you
h
better yet, you can do a hybrid approach... get the features from your own proxy while running a streaming connection for live updates from the GB streaming servers. To do this, pass
streamingHost: <https://cdn.growthbook.io>
into the SDK context (and remove the backgroundSync: false part)
πŸ™Œ 1
❀️ 1
s
nice, that's even better. Thank you again, have a good day
h
no problem, you as well