https://www.growthbook.io/ logo
Join Slack
Powered by
Hi All. Does Growthbook work in a Kubernetes envir...
# ask-questions
b
Hi All. Does Growthbook work in a Kubernetes environment with multiple pods running? We are setting it up in a self hosted environment and it works fine when it's running with one pod. For scalability and fault tolerance we generally like to have at least 2 pods running for all our deployments, with more based on load. In our environment it when we have 2 Growthbook containers running at the same time, the browser gets random 404 errors loading various 
_next/static/css
and 
_next/static/chunks/*
files. Some page refreshes the app works and some it doesn't.
f
Are the two pods running different versions?
b
Hey Jeremy, same exact deployment so each pod should be identical, we just scaled it up to 2 pods.
f
Are you able to hit the API directly on both pods? The response for 
/
on the API (port 3100) should be a JSON object that shows the build version.
b
With 2 pods it gets these at random. With 1 it works fine.
f
The only time I've seen that error happen is in the middle of a deploy when some servers are running the old version and some are running the new one.
b
ok thanks. I'll dig deeper. Just wanted to know if it was setup to work in a multi pod environment. I don't see any errors in the Growthbook logs on either server.
f
Yeah, we designed it to be completely stateless, so it scales horizontally really well
b
Looks like those next.js files it's getting 404 errors are using random names. Is it possible that one pod is generating different file names than the other? Then if the browser loads some pages from one pod and some from another it would generate 404 errors.
f
The filenames are generated when the Docker container is built. So if both are using the exact same image, they should be identical
b
I haven't used next.js much. Is there a dist folder on the server with the generated files. I can check to see what's in there.
I found it 
/usr/local/src/app/packages/front-end/.next/static
Looks like the same files are on both pods. Maybe some issue with out ingress setup. If I figure it out I'll let you know in case somebody else in the community comes across this issue.
actually I take it back. Some of the files are on one pod and some are on the other. For example one of the pods has that first file a84...css and the other has that second file a2...css.
f
Can you look for the 
buildinfo/SHA
file on both pods? That will tell you what build they are running
b
your right, they are different
Copy code
root@dev-growthbook-7fc5fc585f-kb8n5:/usr/local/src/app/buildinfo# cat SHA
979be72b7ae8a3ff78063fd33523d8da11922de9

root@dev-growthbook-7fc5fc585f-zgx97:/usr/local/src/app/buildinfo# cat SHA
1346e59af98ad5f59d0c53420a3eb742cb5db0d2
maybe it's because we are pulling the "latest" image
maybe we should hard code the version number
Yeah I think that must be it. Looks like the two containers pulled two different images
Copy code
image: growthbook/growthbook:latest
imageID: >-
     <docker-pullable://growthbook/growthbook@sha256:dcadca143e3fee377e938388c57c9737c42e0373a103c6c9b8991b1e83063183>
containerID: >-
     <docker://7615d7109cef9acb1e5c9e546ca10041c50fbd4f9fee0d177754a5c007040ea>d

image: growthbook/growthbook:latest
imageID: >-
      <docker-pullable://growthbook/growthbook@sha256:89a53271c27ad73feb19ae5d353307f39ffc5d3ea8fe7b95fd6e6167cefb9d27>
 containerID: >-
      <docker://c90b31defcff2e67d8bd4baad9790945df19790e735216592925f5c3fa6f518>e
f
ya, we do update quite frequently
b
yeah that was it. Sorry for the trouble. I guess if anybody comes across this issue again just tell them to put a specific image tag and not latest 🙂.
170 Views
Open in Slack
PreviousNext
Powered by