https://www.growthbook.io/ logo
Join Slack
Powered by
Hello, everyone. I hope you're doing well. We dec...
# ask-questions
a
Hello, everyone. I hope you're doing well. We decided to split the stats engine deployment apart from the api deployment. We followed this tiny documentation that instructs about which env vars we must configure. However, our api deployment is receiving Unauthorized (No authorization token was found) from the stats engine deployment. I was reading the GrowthBook source code but I couldn't find anything pointing me to the right direction. The code does not really send an authorization token or something. Does anybody have any idea what I might be missing?
b
Hi, some general troubleshooting first: 1. When/if you change 
APP_ORIGIN
and 
API_HOST
, GrowthBook requires a 
JWT_SECRET
to be explicitly set. Make sure the API deployment has 
JWT_SECRET
set to a long random string, and that it matches across all deployments that need to communicate. 2. Ensure both the 
APP_ORIGIN
and 
API_HOST
environment variables are correctly set and that the domains are considered "same site" (same root domain). 3. If your stats engine deployment is communicating with the API using a 
SECRET_API_KEY
environment variable, note that this key defaults to 
readonly
role. If more permissions are needed, you can set 
SECRET_API_KEY_ROLE
accordingly.
a
Hey, August. Thank you so much for your reply. I appreciate it. 1. I have 
JWT_SECRET
properly set for all deployments. 2. Interesting. Now I believe the issue lays in here. 3. I am not using 
SECRET_API_KEY
About origin and host, a quick context on it: We have 3 different deployments: 1. Internal 2. External 3. Frontend Each of them have 
API_HOST
set with a different value. They all have the same 
APP_ORIGIN
though. And I have just realized we did not set 
API_HOST
for the stats engine deployment. However, I am wondering if the stats engine could have its own origin as well.
m
Does each deployment have it's own stats engine or are you trying to share the stats engine across all the 3 deployments?
a
The second option.
m
Is 
PYTHON_SERVER_MODE=true
 set on the stats server? If it's not, the /stats endpoint won't be available and the router would essentially try to serve the other app endpoints which require a JWT token from the client but no JWT token is sent by app backends for /stats requests.
a
Yes. 
PYTHON_SERVER_MODE=true
is set exclusively for the stats engine deployment
m
What's EXTERNAL_PYTHON_SERVER_URL set to on the app backends?
a
It is http://growthbook-stats-engine.growthbook.svc.cluster.local I created a service named 
growthbook-stats-engine
m
And, just to be clear, the stats engine has PYTHON_SERVER_MODE set to "true". Not True or 1 or anything else.
a
Yes, 
"true"
m
What platform is this running in?
a
Platform? hmm, AWS EKS. Is that what you're looking for?
šŸ‘ 1
m
Are you able to verify the env vars on the running pod to be sure PYTHON_SERVER_MODE has been applied?
a
Yes I am. And it is set properly.
Do you think the stats engine returning 401 Unauthorized relies on not having this env var properly set?
I may double check on it
m
The error says no token was found. The token is only used for the app request endpoints, not for the stats endpoint. But the stats endpoint will only run if that env var is true. Without it, /stats wouldn't get served, the router would try to serve up some other endpoint but they all require a token to be verified first. Hence the error no token was found.
The stats engine is assuming the app containers and stats containers are on the same private network so no auth is done between them.
To clarify, the "stats engine" is a full app backend with the /stats endpoint enabled. The theory is that it's actually trying to serve up a default app route.
a
What's exactly a default app route?
m
There is a middleware that checks for the token before allowing any routes to be evaluated. If PYTHON_SERVER_MODE=true, the /stats endpoint is evaluated and served first, before the middleware and app endpoints would be evaluated or served.
Basically if that env var is not true, you're running a regular backend, which checks tokens for ALL routes.
If it's true, you'll serve a /stats endpoint, without requiring a token... or act like an app server if the endpoint is not stats.
āœ… 1
a
k
stands for 
kubectl
That's the change I had to undo in order to get my backend deployments working again.
m
This is on the app servers, not the stats server though right?
a
Yes that's correct. Although, as I said, I have three different deployments. The change in the screenshot was applied to deployment, let's say, A. Deployment B and C did not have the change. However, I was testing the connectivity through deployment A
m
And you're sure the error on the stats engine is from the request coming from app server A?
a
Yes. Because: ā€¢ Only app server A had the 
PYTHON_SERVER_URL
applied; ā€¢ My test was edit an experiment in the frontend (I actually ran the results query) and see the error
Does it make sense?
m
Yeah. Lets see if we can test more directly. Can you run this command on the stats engine?
Copy code
kubectl exec growthbook-stats-engine-67bd784bfc-kfddb -- curl -s -X POST
  <http://localhost:3100/stats> -H "Content-Type: application/json" -d '{}'
This will take out any other variables that could cause an issue. Let's just make sure the stats engine is setup and responding as expected.
a
exec failed: unable to start container process: exec: "curl": executable file not found in $PATH
wget maybe?
m
Sure. Whatever you've got there.
Just want to post an empty payload to that stats endpoint.
If stats handles the request, we'd expect a response error about bad payload.
a
Copy code
root@growthbook-stats-engine-67bd784bfc-kfddb:/usr/local/src/app# wget --post-data='{}' --header 'Content-Type:application/json' -O- <http://localhost:3100/stats>

--2026-03-02 18:38:52--  <http://localhost:3100/stats>
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:3100... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14 [application/json]
Saving to: 'STDOUT'
m
OK, nice. The stats engine is working correctly so this isn't a problem with stats. Next, see you can hit this same endpoint from the app server in the same way, via k exec and wget.
a
Copy code
root@growthbook-fe-internal-56ddbb8896-bhvwz:/usr/local/src/app# wget --method POST --body-data='{}' --header='Content-Type:application/json' <http://localhost:3100/stats>
--2026-03-02 18:42:18--  <http://localhost:3100/stats>
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:3100... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
Different deployment. You might see it at 
growthbook-fe-internal...
It has a different 
APP_ORIGIN
though
m
But, you need to hit /stats on the stats engine, not localhost.
a
Ok
I am going through k8s service
It worked šŸ¤”
m
Can you shut down B and C so only one app server is running and try from the UI?
a
I cant šŸ˜ž If I do so we will have an outage
Actually I can do this in the staging environment
m
OK, how about try the same method with k exec on B and C, verifying each one.
a
Ok
m
Or, yeah, if you have another env to verify the whole setup in.
Why do you think the original error came from the stats engine? If you saw it in the UI, it could have come from the app container.
a
Because I had 
EXTERNAL_PYTHON_SERVER_URL
And I could confirm the request was executed looking at the logs
m
Sure, but the app container would have thrown the missing token error before trying to hit the stats engine.
a
hmmm
m
So, this really may be an issue the app container.
a
m
It would mean the browser didn't send a valid token.
Can you share the error that shows that being the source?
Or... better yet, lets just look forward. Can you give the UI another try and lets see what we get.
a
Yes, but I need to re-do GB_STATS_ENGINE_POOL_SIZE, CRON_DISABLED and EXTERNAL_PYTHON_SERVER_URL and deploy again
Give me five minutes
This button should trigget the stats request, is that correct?
m
I believe so.
a
I was able to wget the stats engine endpoint. Let's see
m
How did the UI work out?
a
No errors šŸ¤”
m
So looks like it's working?
a
Hmm.. There's nothing different compared to last week. One question.. will it work if I set deployments A, B and C to use the stats engine deployment?
m
Isn't that what you've got now?
a
Now. Right now I am testing deployment A only
m
How are you testing only A?
Is it possible you're hitting B or C and they are using their own stats engines?
a
Because deployment A is the only one with a load balancer in front of it
That means I can go at deployment-a.com (for example)
m
got it
So, B and C will be configured the same right?
If so, throw them in.
a
That's the idea
m
Should be fine. /stats engine will be happy to serve all of them.
a
Wondering why I can't see any logs for 
/stats
There are many 
/healthcheck
and 
/api/features/sdk-xxx
logs on the stats engine deployment
Copy code
growthbook {"level":50,"time":1772478989927,"pid":94,"hostname":"growthbook-fe-internal-58f775bcc8-sv25r","dd":{"trace_id":"69a5e2080000000005aa242279c59e42","span_id":"5578905366920162470","service":"growthbook-frontend","version":"0.0.1","env":"production-enduser"},"msg":"Error fetching from stats engine: Stats server errored with: 401 - Unauthorized"}
growthbook {"level":50,"time":1772478989929,"pid":94,"hostname":"growthbook-fe-internal-58f775bcc8-sv25r","dd":

{"trace_id":"69a5e2080000000005aa242279c59e42","span_id":"5578905366920162470","service":"growthbook-frontend","version":"0.0.1","env":"production-enduser"},"err":{"type":"Error","message":"Stats server errored with: 401 - Unauthorized","stack":"Error: Stats server errored with: 401 - Unauthorized\n    at runStatsEngine (/usr/local/src/app/packages/back-end/dist/services/stats.js:90:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async runSnapshotAnalysis (/usr/local/src/app/packages/back-end/dist/services/stats.js:122:23)\n    at async analyzeExperimentResults (/usr/local/src/app/packages/back-end/dist/services/stats.js:436:49)\n    at async ExperimentResultsQueryRunner.runAnalysis (/usr/local/src/app/packages/back-end/dist/queryRunners/ExperimentResultsQueryRunner.js:290:60)\n    at async ExperimentResultsQueryRunner.refreshQueryStatuses (/usr/local/src/app/packages/back-end/dist/queryRunners/QueryRunner.js:250:26)\n    at async Timeout._onTimeout (/usr/local/src/app/packages/back-end/dist/queryRunners/QueryRunner.js:55:38)"},"msg":"Queries succeeded, failed running analysis"}
It is not working yet unfortunately
m
But it was working for A?
a
These logs are from the A deployment
m
When you hit stats from the A server using wget, you didn't get any errors right?
a
That's correct. Actually depending on the parameters I use (
--body-data
or 
--post-data
) I get either 200 or 401. I have never used wget like this before so I might be messing something up
m
How about with:
Copy code
wget --post-data='{}' --header 'Content-Type:application/json' -O-
  <http://growthbook-stats-engine.growthbook.svc.cluster.local/stats>
a
Copy code
--2026-03-02 19:50:45--  <http://growthbook-stats-engine.growthbook.svc.cluster.local/stats>
Resolving growthbook-stats-engine.growthbook.svc.cluster.local (growthbook-stats-engine.growthbook.svc.cluster.local)... fd65:3134:a4e::4:22fb
Connecting to growthbook-stats-engine.growthbook.svc.cluster.local (growthbook-stats-engine.growthbook.svc.cluster.local)|fd65:3134:a4e::4:22fb|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
m
The stats engine doesn't use basic auth. Seems like this response isn't actually coming from the stats engine.
a
Hmm. Let me re-check the k8s service
I think I found a problem with all our k8s services šŸ˜± Let me work on it and get back to you. Thank you so much for your help, Chris. I appreacite
m
Yeah, let me know what you find. šŸ™‚
šŸ‘ 1
Any progress here @adorable-balloon-99556?
a
Yes. Stats engine is running fine without any issues. All 3 deployments went to CPU throttling zero. We are still measuring the impact in latency though. Thank you so much Chris
šŸ‘ 1
20 Views
Open in Slack
PreviousNext
Powered by