#ask-questions
Title
b
bitter-advantage-46115
10/14/2022, 12:33 PMHi All. Does Growthbook work in a Kubernetes environment with multiple pods running? We are setting it up in a self hosted environment and it works fine when it's running with one pod. For scalability and fault tolerance we generally like to have at least 2 pods running for all our deployments, with more based on load. In our environment it when we have 2 Growthbook containers running at the same time, the browser gets random 404 errors loading various
_next/static/css_next/static/chunks/*f
future-teacher-7046
10/14/2022, 12:34 PM
Are the two pods running different versions?
b
bitter-advantage-46115
10/14/2022, 12:35 PMHey Jeremy, same exact deployment so each pod should be identical, we just scaled it up to 2 pods.
f
future-teacher-7046
10/14/2022, 12:37 PM
Are you able to hit the API directly on both pods? The response for
/b
bitter-advantage-46115
10/14/2022, 12:37 PMWith 2 pods it gets these at random. With 1 it works fine.
f
future-teacher-7046
10/14/2022, 12:38 PM
The only time I've seen that error happen is in the middle of a deploy when some servers are running the old version and some are running the new one.
b
bitter-advantage-46115
10/14/2022, 12:41 PMok thanks. I'll dig deeper. Just wanted to know if it was setup to work in a multi pod environment. I don't see any errors in the Growthbook logs on either server.
f
future-teacher-7046
10/14/2022, 12:43 PM
Yeah, we designed it to be completely stateless, so it scales horizontally really well
b
bitter-advantage-46115
10/14/2022, 12:54 PMLooks like those next.js files it's getting 404 errors are using random names. Is it possible that one pod is generating different file names than the other? Then if the browser loads some pages from one pod and some from another it would generate 404 errors.
f
future-teacher-7046
10/14/2022, 12:56 PM
The filenames are generated when the Docker container is built. So if both are using the exact same image, they should be identical
b
bitter-advantage-46115
10/14/2022, 1:11 PMI haven't used next.js much. Is there a dist folder on the server with the generated files. I can check to see what's in there.
I found it
/usr/local/src/app/packages/front-end/.next/staticLooks like the same files are on both pods. Maybe some issue with out ingress setup. If I figure it out I'll let you know in case somebody else in the community comes across this issue.
actually I take it back. Some of the files are on one pod and some are on the other. For example one of the pods has that first file a84...css and the other has that second file a2...css.
f
future-teacher-7046
10/14/2022, 1:41 PM
Can you look for the
buildinfo/SHAb
bitter-advantage-46115
10/14/2022, 1:44 PMyour right, they are different
Copy code
root@dev-growthbook-7fc5fc585f-kb8n5:/usr/local/src/app/buildinfo# cat SHA
979be72b7ae8a3ff78063fd33523d8da11922de9
root@dev-growthbook-7fc5fc585f-zgx97:/usr/local/src/app/buildinfo# cat SHA
1346e59af98ad5f59d0c53420a3eb742cb5db0d2maybe it's because we are pulling the "latest" image
maybe we should hard code the version number
Yeah I think that must be it. Looks like the two containers pulled two different images
Copy code
image: growthbook/growthbook:latest
imageID: >-
<docker-pullable://growthbook/growthbook@sha256:dcadca143e3fee377e938388c57c9737c42e0373a103c6c9b8991b1e83063183>
containerID: >-
<docker://7615d7109cef9acb1e5c9e546ca10041c50fbd4f9fee0d177754a5c007040ea>d
image: growthbook/growthbook:latest
imageID: >-
<docker-pullable://growthbook/growthbook@sha256:89a53271c27ad73feb19ae5d353307f39ffc5d3ea8fe7b95fd6e6167cefb9d27>
containerID: >-
<docker://c90b31defcff2e67d8bd4baad9790945df19790e735216592925f5c3fa6f518>ef
fresh-football-47124
10/14/2022, 2:06 PM
ya, we do update quite frequently
b
bitter-advantage-46115
10/14/2022, 2:11 PMyeah that was it. Sorry for the trouble. I guess if anybody comes across this issue again just tell them to put a specific image tag and not latest 🙂.
4 Views