Hey GrowthBook team - we noticed our mongodb deployment's CPU usage had gotten really high (I think this happened over a few months time), to the point of causing timeouts/aborted requests - I took a look at some of the mongo logs and it seemed to mostly be from queries that agenda makes to find/update jobs. It seems the

eventCreated

jobs weren't being cleaned up properly (or something along these lines):

test> db.agendaJobs.find({name: 'eventCreated'}).count()
5057475

a majority of these had a

nextRunAt

of `null`:

test> db.agendaJobs.find({name: 'eventCreated', nextRunAt: null}).count()
5056482

I ended up deleting these to see if it helped and our mongo CPU usage dropped pretty much back down to 0. I haven't dug into why so many are created or whether something should actually be cleaning them up, but just thought I'd share here in case this is a known issue. It's possible its related to our own setup, although I don't think we have any changes that would affect this job type specifically

Thanks, I think we might be missing some indexes on that collection. So those un-indexed queries were doing full scans which get really expensive as the size grows. In addition to adding missing indexes, we should be able to add a cronjob to clean up those old jobs pretty easily.

my not-so-comfortable-with-node/typescript guess is that it's because

agenda.define

is used in the constructor of the `EventNotifier`: https://github.com/growthbook/growthbook/blob/1a9e129bfae401e62bde002fb4679b5dc9de956d/packages/back-end/src/events/notifiers/EventNotifier.ts#L[…]6 and in all usages of

perform()

a new EventNotifier is created

creating new docs for every event is the proper behavior since the event id is part of the unique key. We're just missing the cleanup part

got it, that makes sense

Added a GitHub issue to track this - https://github.com/growthbook/growthbook/issues/2116