I have set the following 2 env vars (the process in running on

k8s

)

OTEL_EXPORTER_OTLP_ENDPOINT: <http://prometheus-pushgateway:9091/metrics>
        OTEL_SERVICE_NAME: growthbook

Our

prometheus-pushgateway

is running and is exposed as an internal k8s service. From within a growthbook pod:

curl <http://prometheus-pushgateway:9091/metrics>
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
.
.
.

However, what I am seeing in the

growthbook

logs is the following

Process is not running on K8S Error: Failed to load page, status code: 403
    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/resource-detector-aws/build/src/detectors/AwsEksDetector.js:192:32)
    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:319:14)
    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

{
  "stack": "OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n    at AsyncLocalStorage.run (node:async_hooks:319:14)\n    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)",
  "message": "Not Found",
  "name": "OTLPExporterError",
  "data": "404 page not found\n",
  "code": "404"
}

Any suggestions?

You should not include the

/metrics

part in your endpoint. That is added automatically by the OpenTelemetry library we're using.

I am getting the exact same behaviour with this

Hmm. Looks like there may be some issues with using prometheus pushgateway with Open Telemetry. https://github.com/open-telemetry/opentelemetry-js/issues/1278

ah ok. so the opentelemetry exporter library used in GB cannot use prometheus pushgateway as

OTEL_EXPORTER_OTLP_ENDPOINT

if I understand correctly?

That's what it looks like. I haven't tested this myself, so not 100% sure. I have tested with the OTEL collector - https://opentelemetry.io/docs/collector/ although I was sending to different destinations, not Prometheus. They do list Prometheus as supported there, so it will likely work.

hm ok. if anything comes up on this do let us know, it would be highly appreciated