https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
b

bitter-advantage-46115

10/14/2022, 12:33 PM
Hi All. Does Growthbook work in a Kubernetes environment with multiple pods running? We are setting it up in a self hosted environment and it works fine when it's running with one pod. For scalability and fault tolerance we generally like to have at least 2 pods running for all our deployments, with more based on load. In our environment it when we have 2 Growthbook containers running at the same time, the browser gets random 404 errors loading various
_next/static/css
and
_next/static/chunks/*
files. Some page refreshes the app works and some it doesn't.
f

future-teacher-7046

10/14/2022, 12:34 PM
Are the two pods running different versions?
b

bitter-advantage-46115

10/14/2022, 12:35 PM
Hey Jeremy, same exact deployment so each pod should be identical, we just scaled it up to 2 pods.
f

future-teacher-7046

10/14/2022, 12:37 PM
Are you able to hit the API directly on both pods? The response for
/
on the API (port 3100) should be a JSON object that shows the build version.
b

bitter-advantage-46115

10/14/2022, 12:37 PM
With 2 pods it gets these at random. With 1 it works fine.
f

future-teacher-7046

10/14/2022, 12:38 PM
The only time I've seen that error happen is in the middle of a deploy when some servers are running the old version and some are running the new one.
b

bitter-advantage-46115

10/14/2022, 12:41 PM
ok thanks. I'll dig deeper. Just wanted to know if it was setup to work in a multi pod environment. I don't see any errors in the Growthbook logs on either server.
f

future-teacher-7046

10/14/2022, 12:43 PM
Yeah, we designed it to be completely stateless, so it scales horizontally really well
b

bitter-advantage-46115

10/14/2022, 12:54 PM
Looks like those next.js files it's getting 404 errors are using random names. Is it possible that one pod is generating different file names than the other? Then if the browser loads some pages from one pod and some from another it would generate 404 errors.
f

future-teacher-7046

10/14/2022, 12:56 PM
The filenames are generated when the Docker container is built. So if both are using the exact same image, they should be identical
b

bitter-advantage-46115

10/14/2022, 1:11 PM
I haven't used next.js much. Is there a dist folder on the server with the generated files. I can check to see what's in there.
I found it
/usr/local/src/app/packages/front-end/.next/static
Looks like the same files are on both pods. Maybe some issue with out ingress setup. If I figure it out I'll let you know in case somebody else in the community comes across this issue.
actually I take it back. Some of the files are on one pod and some are on the other. For example one of the pods has that first file a84...css and the other has that second file a2...css.
f

future-teacher-7046

10/14/2022, 1:41 PM
Can you look for the
buildinfo/SHA
file on both pods? That will tell you what build they are running
b

bitter-advantage-46115

10/14/2022, 1:44 PM
your right, they are different
Copy code
root@dev-growthbook-7fc5fc585f-kb8n5:/usr/local/src/app/buildinfo# cat SHA
979be72b7ae8a3ff78063fd33523d8da11922de9

root@dev-growthbook-7fc5fc585f-zgx97:/usr/local/src/app/buildinfo# cat SHA
1346e59af98ad5f59d0c53420a3eb742cb5db0d2
maybe it's because we are pulling the "latest" image
maybe we should hard code the version number
Yeah I think that must be it. Looks like the two containers pulled two different images
Copy code
image: growthbook/growthbook:latest
imageID: >-
     <docker-pullable://growthbook/growthbook@sha256:dcadca143e3fee377e938388c57c9737c42e0373a103c6c9b8991b1e83063183>
containerID: >-
     <docker://7615d7109cef9acb1e5c9e546ca10041c50fbd4f9fee0d177754a5c007040ea>d

image: growthbook/growthbook:latest
imageID: >-
      <docker-pullable://growthbook/growthbook@sha256:89a53271c27ad73feb19ae5d353307f39ffc5d3ea8fe7b95fd6e6167cefb9d27>
 containerID: >-
      <docker://c90b31defcff2e67d8bd4baad9790945df19790e735216592925f5c3fa6f518>e
f

fresh-football-47124

10/14/2022, 2:06 PM
ya, we do update quite frequently
b

bitter-advantage-46115

10/14/2022, 2:11 PM
yeah that was it. Sorry for the trouble. I guess if anybody comes across this issue again just tell them to put a specific image tag and not latest 🙂.
37 Views