https://www.growthbook.io/ logo
Join Slack
Powered by
My bad. You said you were using GBC. Is the proxy ...
# ask-questions
a
My bad. You said you were using GBC. Is the proxy on kubenetes or Docker
s
no, Kubernetes Secret was not recently updated
Could you please provide a guide on how to configure proper logging for the GB Proxy? We currently don’t see SSE interactions with GB Server or errors in the logs.
a
I think that's doable. First, share your entire stack, i.e. what is sitting between the SDK/browser and the GBProxy and between the proxy and GB Cloud
Also, were you able to determine if Pod A works, Pod B returns 403, Pod N has stale config This would cause recheck to appear to fix it because the next request lands on Pod A.
s
we have fixed this issue. it was because of WAF rules
🙌 1
but we still need help to configure logs for GB Proxy. Cause for now we don't see interactions (logs) via SSE
for GB Proxy - GB Cloud Server and Mobile SDK - GB Proxy
a
Ok. You are in PROD at the moment?
s
yes
We want to be sure that interactions happen and to check the data that we send via SSE
and check what config is on GB Proxy for particular POD
to be sure that everything works as we expect
a
SSE does not send user interaction events to GrowthBook. SSE is the server-client stream that sends feature/experiment definition updates from GrowthBook Cloud/Proxy down to the SDK. GrowthBook’s docs describe it as streaming feature changes to the SDK in realtime after they are published (https://docs.growthbook.io/lib/js?utm_source=chatgpt.com#streaming-updates)
If you review where GB is initialized in your mobile sdk, does it include 
streaming: true
s
SSE works for us. but we want more observability and alerts when something happens
a
Sure that makes sense. What types of system analytics tools are currently deployed?
s
Big Query
but we don' connect it to GB Server
for now we use GB only to get experiment variation or feature flag
a
... and it sounds like you need more data than is currently available oob
if this is correct, will you be using SQL to analyze the data?
s
It’s not our main problem right now. For now, the GB Proxy and Cloud Server are black boxes to us because we don’t have sufficiently informative logs.
and can’t do anything proactively if something happens
a
Well with logging, as shared earlier, depends on the type of logs youre after. It's also a config. We asked about PROD earlier to determine if it was wise to turn on 
development
.
NODE_ENV=production
is used to hide debug and informational log messages
s
NODE_ENV=development VERBOSE_DEBUGGING=true
• This is our current production configuration
a
Copy code
yaml (your Kubernetes patch)

env:
  - name: NODE_ENV
    value: development

(then restart, example)
kubectl rollout restart deployment growthbook-proxy -n growthbook
kubectl logs -n growthbook deploy/growthbook-proxy -f
s
@green-lighter-44664 could you please provide our current config yaml
g
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: growthbook-proxy
  namespace: growthbook
spec:
  replicas: 2
  selector:
    matchLabels:
      app: growthbook-proxy
  template:
    metadata:
      labels:
        app: growthbook-proxy
    spec:
      containers:
        - name: proxy
          image: growthbook/proxy:v1.3.3
          ports:
            - containerPort: 3300
              protocol: TCP
          env:
            - name: GROWTHBOOK_API_HOST
              value: <https://api.growthbook.io>
            - name: NODE_ENV
              value: development
            - name: SECRET_API_KEY
              valueFrom:
                secretKeyRef:
                  name: growthbook-proxy-secret
                  key: secret_api_key
            - name: PUBLISH_PAYLOAD_TO_CHANNEL
              value: 'true'
            - name: CACHE_ENGINE
              value: redis
            - name: CACHE_CONNECTION_URL
              value: >-
                <redis://growthbook-proxy-redis-master.growthbook.svc.cluster.local:6379>
            - name: OTEL_SERVICE_NAME
              value: growthbook-proxy
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: http/protobuf
            - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
              value: >-
                <http://opentelemetry-collector.otel-collector.svc:4318/v1/metrics>
            - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
              value: <http://opentelemetry-collector.otel-collector.svc:4318/v1/logs>
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: <http://opentelemetry-collector.otel-collector.svc:4318/v1/traces>
            - name: OTEL_METRICS_EXPORTER
              value: otlp
            - name: OTEL_LOGS_EXPORTER
              value: otlp
            - name: OTEL_TRACES_EXPORTER
              value: otlp
            - name: OTEL_PROPAGATORS
              value: tracecontext,baggage
            - name: OTEL_BLRP_MAX_QUEUE_SIZE
              value: '1000000'
            - name: OTEL_BLRP_MAX_EXPORT_BATCH_SIZE
              value: '1024'
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: k8s.namespace.name=growthbook,deployment.environment=prod
            - name: OTEL_NODE_RESOURCE_DETECTORS
              value: env,host,os,process,serviceinstance
            - name: OTEL_BSP_MAX_QUEUE_SIZE
              value: '4096'
            - name: OTEL_EXPORTER_OTLP_TIMEOUT
              value: '30000'
            - name: VERBOSE_DEBUGGING
              value: '1'
          resources: {}
          livenessProbe:
            tcpSocket:
              port: 3300
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            tcpSocket:
              port: 3300
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
👍 2
a
and the Service YAML?
g
Copy code
apiVersion: v1
kind: Service
metadata:
  name: growthbook-proxy
  namespace: growthbook
  uid: db2fc3a5-d3b1-4a58-8b27-3bf740318562
  resourceVersion: '161601000'
  creationTimestamp: '2026-03-17T14:43:02Z'
  annotations:
    <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"growthbook-proxy","namespace":"growthbook"},"spec":{"ports":[{"port":3300,"protocol":"TCP","targetPort":3300}],"selector":{"app":"growthbook-proxy"},"type":"ClusterIP"}}
  selfLink: /api/v1/namespaces/growthbook/services/growthbook-proxy
status:
  loadBalancer: {}
spec:
  ports:
    - protocol: TCP
      port: 3300
      targetPort: 3300
  selector:
    app: growthbook-proxy
  clusterIP: 10.100.152.63
  clusterIPs:
    - 10.100.152.63
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster
👍 2
a
If the proxy is up and running, can we try some troubleshooting?
s
yes. what do we need to do?
a
Preliminary ingress check: 
kubectl get ingress -n growthbook
wait let's back up. you stated earlier that you discovered the firewall rule was causing the initial issue with the intermittent 403s, correct?
so in terms of the logging, you aren't interested in interactions so much as logging at different layers of your GB stack. does that sound about right?
from https://github.com/growthbook/growthbook-proxy/blob/main/README.md?plain=1 "The GrowthBook Proxy is instrumented with OpenTelemetry to publish observability metrics, traces, and logs. To enable, you must change the Docker CMD from the default 
pnpm start
to 
pnpm start:with-tracing
."
try this:
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: growthbook-proxy
  namespace: growthbook
spec:
  replicas: 2
  selector:
    matchLabels:
      app: growthbook-proxy

  template:
    metadata:
      labels:
        app: growthbook-proxy

    spec:
      containers:
        - name: proxy
          image: growthbook/proxy:v1.3.3
          imagePullPolicy: IfNotPresent

          # Critical for GrowthBook Proxy OpenTelemetry support
          command: ["pnpm"]
          args: ["start:with-tracing"]

          ports:
            - name: http
              containerPort: 3300
              protocol: TCP

          env:
            - name: GROWTHBOOK_API_HOST
              value: <https://api.growthbook.io>

            - name: NODE_ENV
              value: production

            - name: SECRET_API_KEY
              valueFrom:
                secretKeyRef:
                  name: growthbook-proxy-secret
                  key: secret_api_key

            # Multi-replica cache / pub-sub sync
            - name: PUBLISH_PAYLOAD_TO_CHANNEL
              value: "true"

            - name: CACHE_ENGINE
              value: redis

            - name: CACHE_CONNECTION_URL
              value: <redis://growthbook-proxy-redis-master.growthbook.svc.cluster.local:6379>

            # Optional, but useful if you are using SSE / streaming
            - name: ENABLE_EVENT_STREAM
              value: "true"

            # OpenTelemetry service identity
            - name: OTEL_SERVICE_NAME
              value: growthbook-proxy

            - name: OTEL_RESOURCE_ATTRIBUTES
              value: k8s.namespace.name=growthbook,service.namespace=growthbook,deployment.environment=prod

            - name: OTEL_NODE_RESOURCE_DETECTORS
              value: env,host,os,process,serviceinstance

            # OTLP over HTTP/protobuf
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: http/protobuf

            - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
              value: <http://opentelemetry-collector.otel-collector.svc:4318/v1/metrics>

            - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
              value: <http://opentelemetry-collector.otel-collector.svc:4318/v1/logs>

            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: <http://opentelemetry-collector.otel-collector.svc:4318/v1/traces>

            # Enable exporters
            - name: OTEL_METRICS_EXPORTER
              value: otlp

            - name: OTEL_LOGS_EXPORTER
              value: otlp

            - name: OTEL_TRACES_EXPORTER
              value: otlp

            - name: OTEL_PROPAGATORS
              value: tracecontext,baggage

            # Export tuning
            - name: OTEL_BSP_MAX_QUEUE_SIZE
              value: "4096"

            - name: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
              value: "512"

            - name: OTEL_BLRP_MAX_QUEUE_SIZE
              value: "4096"

            - name: OTEL_BLRP_MAX_EXPORT_BATCH_SIZE
              value: "512"

            - name: OTEL_EXPORTER_OTLP_TIMEOUT
              value: "30000"

            # Use temporarily during setup; remove or set to 0 after validation
            - name: VERBOSE_DEBUGGING
              value: "1"

          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi

          readinessProbe:
            httpGet:
              path: /healthcheck
              port: 3300
            initialDelaySeconds: 5
            timeoutSeconds: 2
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /healthcheck
              port: 3300
            initialDelaySeconds: 15
            timeoutSeconds: 2
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3

          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File

      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%

  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
g
we are receiving this on pod startup
Copy code
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "pnpm": executable file not found in $PATH. (8s)
a
ok. does the error continue when using 
npm
in place of 
pnpm
g
it works with that command and args:
Copy code
command: ["npm"]
args: ["run", "start:with-tracing"]
a
ok. so you are getting the logging you were looking for?
g
@silly-engine-11095 please check
a
well let me know once you have a chance to confirm log generation
s
We can see the logs, but they aren’t human-readable. Everything is stored in the response body as a collection of traces
It seems that the problem is that GrowthBook does not send logs to OTEL. OTEL collects them from the pod. We need GrowthBook to send the logs directly somehow.
a
ok. just to clarify, the changes you made are now yielding traces and logs
want to make sure we're progressing vs regressing
s
we still on the same page, cause seems with
Copy code
command: ["npm"]
args: ["run", "start:with-tracing"]
we only have " | " as a first symbol
left screen - before, right - after
a
and possibly emitting from a different pod
Have you discussed using an OTEL Collector with an OTEL oltp endpoint?
Are you considering using OpenTelemetry to capture the following? • experiment exposure • experiment key • variation ID/value • feature key • user/session ID, ideally hashed • targeting attributes • cart_open or other runtime attributes • conversion events • request traces correlated with variation • errors correlated with variation • latency correlated with variation
s
https://growthbookusers.slack.com/archives/C01T6Q1SVFV/p1777488683195009?thread_ts=1777065093.677939&amp;cid=C01T6Q1SVFV We tried just that, but for some reason, Grossbook doesn't send logs to the hotel via the otlp endpoint.
https://growthbookusers.slack.com/archives/C01T6Q1SVFV/p1777495302073219?thread_ts=1777065093.677939&amp;cid=C01T6Q1SVFV We want to monitor SSE interactions between the GB Proxy and the server, as well as between clients and the GB Proxy, and, of course, capture any errors if they occur. We want to make sure SSE is working reliably for us.
a
Please confirm that the collector is running on one of your pods
Might you be looking for something as follows:
9 Views
Open in Slack
PreviousNext
Powered by