https://www.growthbook.io/ logo
#announcements
Title
# announcements
b

brief-traffic-36675

08/17/2023, 3:02 PM
I have set the following 2 env vars (the process in running on
k8s
)
Copy code
OTEL_EXPORTER_OTLP_ENDPOINT: <http://prometheus-pushgateway:9091/metrics>
        OTEL_SERVICE_NAME: growthbook
Our
prometheus-pushgateway
is running and is exposed as an internal k8s service. From within a growthbook pod:
Copy code
curl <http://prometheus-pushgateway:9091/metrics>
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
.
.
.
However, what I am seeing in the
growthbook
logs is the following
Copy code
Process is not running on K8S Error: Failed to load page, status code: 403
    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/resource-detector-aws/build/src/detectors/AwsEksDetector.js:192:32)
    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:319:14)
    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Copy code
{
  "stack": "OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n    at AsyncLocalStorage.run (node:async_hooks:319:14)\n    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)",
  "message": "Not Found",
  "name": "OTLPExporterError",
  "data": "404 page not found\n",
  "code": "404"
}
Any suggestions?
f

future-teacher-7046

08/17/2023, 3:56 PM
You should not include the
/metrics
part in your endpoint. That is added automatically by the OpenTelemetry library we're using.
I think it's trying to hit
/metrics/metrics
now and that's why it's throwing a 404 error
b

brief-traffic-36675

08/17/2023, 3:58 PM
I am getting the exact same behaviour with this
Copy code
root@growthbook-api-web-c4f677c8-wh749:/usr/local/src/app# env | grep -i otel
OTEL_EXPORTER_OTLP_ENDPOINT=<http://prometheus-pushgateway:9091>
f

future-teacher-7046

08/17/2023, 4:03 PM
Hmm. Looks like there may be some issues with using prometheus pushgateway with Open Telemetry. https://github.com/open-telemetry/opentelemetry-js/issues/1278
b

brief-traffic-36675

08/17/2023, 4:08 PM
ah ok. so the opentelemetry exporter library used in GB cannot use prometheus pushgateway as
OTEL_EXPORTER_OTLP_ENDPOINT
if I understand correctly?
f

future-teacher-7046

08/17/2023, 4:11 PM
That's what it looks like. I haven't tested this myself, so not 100% sure. I have tested with the OTEL collector - https://opentelemetry.io/docs/collector/ although I was sending to different destinations, not Prometheus. They do list Prometheus as supported there, so it will likely work.
b

brief-traffic-36675

08/17/2023, 4:13 PM
hm ok. if anything comes up on this do let us know, it would be highly appreciated
FWIW, the exception
Process is not running on k8s
seems to be related to a check of whether the app is running specifically on EKS. I have no experience on
node.js
but could the library be useful only on EKS? (don't think so but just asking)
14 Views